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FOREWORD 



The ARI Fort Knox Field Unit has for several years_ been involved in the 
application of behavioral science to increasing the quality of the products of 
Army Centers/Schools. These products include training, programs, training sup- 
port materials^ _and^ ultimately, trained soldiers who have participated in the 
programs or utilized the materials. A crucial aspect of quality control of 
these products is the gathering of valid and reliable feedback to support prod- 
uct evaluation and redesign. _ _______ 

For approximately three years, the Training Evaluation/Feedback Team of 
the Fort Knox Field Unit has addressed the collection arid use of feedback by 
Center/ School personnel. This 'effort has focused upon the design of compre- 
hensive but practical systems for gathering feedback, arid it has included sev- 
eral applications of the methods developed. The experiences gained during this 
applied research serve as the basis for this handbook. 

The present report is designed to serve as a practical gu ide for the col- 
lect ion and management of feedback. It presents flexible guidelines which can 
be tailored to meet the needs of each f eedback situation. v It is intended pri- 
marily to serve the needs of training developers and eva luators within TRADOC 
Centers/Schools , but it should be of use to all who are involved in the evalu- 
ation of training. 

/// .> 
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EDGAR M. JOHNSON__ 
Technical Director 



TRAINING FEEDBACK HANDBOOK 
BRIEF 



Requirement ^ evaluat0 rs need feedback in order to determine 

whether products of the training development process meet the needs of users. 
This handbook is designed to assist these personnel in structuring their col- 
lection and use of feedback. 

Procedure* - - . 

The numerous methods for collecting feedback are grouped into six cate- 
gories in this report: informal feedback, existing unit performance records, 
questionnaires, structured interviews, systematic observation, and -" s ^|' 
lased upon available literature and previous research experience, the positive 
and negative aspects of each of these methods are discussed and general/ guide- 
lines are offered for the application of eac* method. Other issue s a ^ssed^. 
include the integration of methods and the management and analysis of /feedback. 



Fl No one method is Sufficient in and of itself for collection of /feedback 
in all situations. But use of a mix of methods and tailored application of 
the Guidelines offered in this handbook should provide the feedback that train- 
ing developers and evaluators need. Integration of data to insure their accu- 
racy, automation of data management, and follow-up to insure that data have 
the appropriate impact are key issues in the use of feedback. < 

Util 1KfLndboofwifi : be useful to all training developers and e.aluators in 
their collection of feedback from personnel in field units and in Centers/ 
Schools. It will also have general utility for all personnel involved in the 
evaluation of training. 
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SECTION I - INTRODUCTION 

In the context of Army trainings feedback is any information _abbut the 
results of the training development and implementation process. This informal 
tibri may take various forms , such as the results of tests administered to per- 
sonnel enrolled in a training program, observations of training exercises, or * 
opinions of participants in the training process. But regardless of its form, 

the purpose of feedback is the same to provide input to .evaluation of the 

effectiveness and efficiency of training. Analysis of feedback indicates 
whether or not the training process has achieved its goals. If the goals have 
not been met, then feedback should indicate the changes that are needed in 
training. Feedback is thus crucial to the training process; without it, there 
is no way of determining when training is effective. » 

Feedback is needed by all participants in the training process. Trainers 
and trainees need feedback to determine when training has been effective and 
when further training is needed. Training managers and commanders need feed- 
back to determine the training readiness of their units and to schedule and 
provide resources for needed training; Training developers and evaluators in 

..TRADQC Center ,sZSchopls^aeed^eedback„f rom users^o^eLermin£L_^&rJier the 

school's products have resulted in effective training, and to make modifications 
as needed. It is not possible to directrix address - the f eBdback'^eeds"of T all 
these personnel in this handbook. Therefore, the emphasis here is upon the 
needs of one speciiic group, training developers and evaluators in TRADOC Cen- 
ters/Schools. But the guidance. and methods presented should be of general use 
to all individuals involved in the training process, including \thpse managing 
training in field units. j 

This handbook is designed to assist training developers and evaluators in 
structuring their collection and use of feedback. Training development and 
evaluation are intertwined processes , but in most TRADOC Centers/Schools the 
personnel performing them are divided into two separate directorates. This is 

done to insure that personnel are riot placed in the potentially compromising 

position of haying to evaluate their own products. Training developers are per- 
sonnel who produce programs or materials that are utilized to directly or indi- 
rectly support the training of soldiers. Here the term refers generally to 
personnel assigned to the Directorates of Training Developments (DTD's) of 
TRADOC Centers/Schools ^ but it may also include personnel in instructional de- 
partments arid other agencies who are involved in the training development pro- 
cess. The term training evaluators is used here to refer generally to personnel 
assigriedtb the Directorates of Evaluation and Standardization (DOES's) of 
Centers/Schools, although it may refer to anyone wItlo determines the effectiveness 
of training development arid implementation. In t he -t-y pica i Cen t er / S c ho ol,— train - 
irig evaluators have the primary responsibility for bollecting feedback and pro- 
viding it to training developers and others having a need for it. 

As stated earlier, feedback relates to the results of training development 
and imp 1 emen tat ion . From the perspective of training developers and evaluators, 
the results of training development are training programs and training support 
materials, and the results of implementation are the states of individual and 
collective training achieved through use of these products. Training programs 
are structured sets of procedures, whose primary purpose is to provide needed 
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skills arid knowledge to soldiers. Examples are basic entry training, basic and 
advanced NCO and officer courses, and training exercises derived from the Army 
Training and Evaluation Program (ARTE?) . '-training support materials are prod- 
ucts of the training development process which enable training programs to be 
conducted. Included here are written manuals and guides, audiovisual materials, 
results of front-end analyses, ARTEP documents, and all other materials whose 
pr imary use is to support the -training of soldiers . — - - ~ 

Iri order to fully evaluate the effectiveness of training programs arid 
training support materials^ feedback is needed from all the users of these 
products. These users include commanders, training managers, trainers, and 
trainees, both iri the institution arid in operational field units. Feedback from 
personnel within the institution is designated internal^ arid feedback from per- 
sonnel in field units _is designated external. These two classifications of 
feedback are discussed jointly iri this handbook, since the same general methods 
arid guidelines apply to both. 



Feedback from users should riot be based largely on their opinions about 
p rbduc ts-ns f— -the— tr ain in g development- process,— but -should- in st ead- r epxesenx^c rb^^ 
jective results of their experiences with these products. The ultimate purpose 

of t rain in g - d e velopmen t products. JLs to enable training to be provided to sol- . 

diers, and thereby to increase the combat readiness of the Army. Feedback 
should thus communicate the levels of training readiness resulting from the 
implementation of these products. It should indicate which products or parts of 

products have resulted^ in effective training, 

where effective training has not occurred, feedback should be detailed enough to 
allow determination of whether the proplem lies in the product itself or in the 
way in which it was implemented. Corrective action can then be directed at the 
appropriate point in the training development and implementation process. 

While feedback based upon the results of implementation of products isof 
prime importance, feedback is needed during all phases of the training develop- 
ment process. Training in the Army is based upon the Instructional Systems 
Development (ISD) model, which consists of five phase? : evaluation, analysis* _ 
design, development, and implementation. Evaluation is riot a separate independ- 
ent phase in this model, but is rather a process which should be integrated into 
ail the other phases. Feedback", or the data utilized in evaluation -> can thus be 
useful at any point iri the training development process.. The decision to initi- 
ate the analysis arid design phases may be based upon feedback indicating that 
new training programs or support materials are needed. Internal feedback 
gathered durj.rigt;^ — 
priate tasks were not selected for training during the analysis phase, and the 
analysis may thus have to be( revised. Feedback should flow continually through- 
out the training development process arid not be gathered at just one point in 
time. In this handbook emphasis is placed upon feedback gathered during and 
after the imp 1 emen t a t i on of products, but the methods and guidelines can and 
should be. applied during all phases of the training development process. 

The first step in the feedback process is to determine what information is 
needed. While the specific needs will vary from situation to situation, the 
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general characteristics needed in feedback can be summarized here. Training 
developers need feedback which is specific to each of _their products. They 
need _ data irldicatin S the proficiencies of soldiers arid units who have partic- 
ipated in training programs or used training support materials* arid informa- 
tion on how the programs were conducted or how the materials were used. They 
nee ^_ to _] aiow w frich materials or parts of materials have been used in field 
units; and which had to^ be 'modified before they could be used. They need to 
know whe t he r t he, tasks , condition s , . s t an d a r d s , an d re sour e e requiremerits 
included in training programs and materials have proved to be realistic. 
Training developers and evaluators basically need all the feedback they can 
get, as long as it is manageable and related to specific products. Training 
developers within Gentexs/Schoois are sometimes i isolated from the realities 
of the field unit environment. They need to knqw what is happening in the 
field, and the collection of feedback and contacts_ with f ield personnel are 
the best ways to meet this need; 

As indicated above* training developers and evaluators need many differ- 
ent types of feedback. They should obtain a variety of complementary data 
from different sources , in order tb insure the accuracy of feedback. There are 
also numerous characterist ics which e ach ind ividual t ype of fee dback should 
haveT" Feedback should be as specific and detsile^Liak^ossible ; it should relate 
to the performance of specif ic tasks or the use^of specific Mparts" of training" 
materials. This is necessary so that the identification of problems in feedback 
leads to possible solutions. General feedback such as "lieutenants can't 'read 
maps 11 or "mechanics can't troubleshoot" is not of much use to training develop- 
ers. In order to modify training programs arid materials, the developers need 
more specific information detailing which personnel" are~having difficulty with" 
which aspects of which tasks. Feedback should also be objective or based on 
observable evidence, rather than subjective or, tased just oh ^pinions. Guide- 
lines such as those offered in this handbook should be followed to maximize the 
accuracy of all feedback collected. Feedback should also be timely arid repre- 
sentative of the current situation iri the field. In today f s tap idly changing 
state of technology , the situatidri iri the field can change quickly arid feedback, 
processes should be attuned to this. Depending on the situatidri, feedback may 
need\ to be collected on each of several occasions, in order td insure that it is 
up-to-date. Since a high volume of feedback is needed, it should be quantifi- 
able or easily encoded so that it can be entered into. automated data processing 
systems. No feedback should be collected without firm plans as to how it will 
be analyzed, managed^ and used. Use of this handbook-^will not guarantee ful- 
fillment of all the feedback needs discussed above. But sound refinement and 
tailoring of the principles offered in it should help structure feedback to 
mee t t he need s of. . t he Army . 

One characteristic of feedback which deserves special mention here is the 
need for it to be coordinated and integrated. One aspect of this is the need 
for training developers and evaluators to communicate continually. As was 
pointed out earlier, training developers and evaluators at most TRABOC Centers/ 
Schools are divided into two directorates , • the \DTD and the DOES. Some inde- 
pendence is needed between these two seLs_ of personnel , in __order„ toziessen -Lhe— 

-probability- of ~bias~dur Ing" t^fie evaluation of products, but they should 
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constantly communicate with each other as to what feedback has;.beeri collected and 
what further feedback is needed. The feedback process should be a continuous 
loop, with training developers identifying what feedback is^needed^ training 
evaluators collecting it arid providing it to those who need it and training de- 
velopers indicating what was done in response to the feedback arid, what new feed- 
back is needed. Training evaluators gather feedback during Branch Training Team 
(BTT) visits or other visits to field units. They should involve training devel- 
opers in planning these visits and'speciiying" tfie feedback to _be collected. A 
general telephonic or written request to training developers for questions to be : 
asked during upcoming visits may not l>e sufficient. Face-tb-face active coordi- 
nation between training developers and evaluators is needed prior to feedback ^ 
collection visits. Consideration should be given to continuing this coordination 
throughout field visits by including training developers on data collection teams. 
In this way training developers may-be -bet ter able to collect- the detailed feed- 
back that they heed and may have more confidence iri^the data^ collected. Coordina- 
tion should :klso continue during analysis and use of feedback, in order to lessen 
the probability of differing interpretations of the data collected and to insure 
a common understanding of the feedback needed during future field visits. , These 
cont-tauous-coordination^ of effort by train- / 

ing developer^ and evaluators and should insure that feedback meets the needs of \ 
all concerned. \ I 

Another aspect of the coordination of feedback involves the integration of / 
the methods used to collect it. As will be discussed in detail in the next sec- \ 
tion, there are several methods through which feedback can be collected, and none 
of them are sufficient for all situations. A mix of methods must thus be tailored 
to the resources available and the scope of the problems encountered. For exam- 
ple, feedback bri a problem with a training product may first be provided to Cen- 
ter/School personnel through informal communications from the field. The problem 
can then be further examined through administration of surveys or interviews to 
field personnel. If the problem appears to be a severe one, it may then be nec- 
essary to address it further through structured observations or testing in the 
field. The integration of different methods may also increase the accuracy of 
feedback^ since the data collected from different approaches or sources can be 
used as cross-checks. Using the appropriate mix of methods in collecting feed- 
back is. an important concern, and a separate section will be devoted to it in 
this handbook. 

The final aspect of the coordination of feedback tc be introduced here in- 
volves the integration of the data collected. All the data relating to a partic- 
ular product or issue should be managed so that theyXcari be used in i s complemen- 
tary fashion. Feedback should be maintained in a centralized data base which is 
accessible to both trailing developers and evaluators.' In this way duplications 
of effort can be reduced and decisions to .revise training products can be based 
upon all the data available. Guidelines for the management and use of feedback 
will be presented in later sections of this handbook. 



The collection and use of feedback is a continuous adaptive process, *nd_ 
there is /no unique solution or set procedure for accomplishing it ._ Thus, wh i le . 
the goal of this handbook is to increase the structure and standardization^ of the 
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flow of feedback to- and within TRADOC Centers/Schools, it present??, general 
guidelines rather than rigid procedures to be followed. General "how to 1 ' pro- 
cedures are presented which training developers and evaluators can tailor to 
their individual organizational needs and available resources.. Included in 
these procedures are descriptions of the primary methods which can be used _ to 
collect feedback, discussions of general approaches to analyzing and managing 
the data collected, and. suggestions for ways to utilize feedback to improve 
the products evaluated. 
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SECTION II - FEEDBACK METHODS 



There are tnan£ ways in which training developers arid evaluatbrs can col- 
lect feedback relating to products of the^ttata Ex- 
ternal feedback can be gathered through formal visits to field units by Branch 
Training Teams (BTTs) or other groups of Center/School personnel ^ through com- 
munications with field personnel using t el ephories_br the mail, or through 
interaction with field personnel when they^are temporarily available at the 
Center /School. Internal feedback can be gathered through visiting institu- 
tional training sites or- through interactions with trainers arid trainees. 
Several general administrative issues arise during planning for the collection 
of feedback, such as who should collect it and how often it should be collec- 
ted. Such general issues are riot directly addressed in this handbook, except 
... in terms, of broad^gui<Je±^e_s^^^ focuses 

upon relatively specific' feedback collection methods which can be applied 

either at the. institution or in the \ field. Managers of BTT's or other feedback 
collection efforts should integrate these methods and tailor the guidelines 
offered to meet th,^ heeds of each particular /situation. In this way , structure 
can be introduced into the feedback process while maintaining its needed flexi- 
bility. 

The feedback methods discussed in this chapter are organized into six 
— ca€ij5ffelt informal feedback, existing unit performance records, question- 
naires v structured -toterv-iew^ and testing. Each of _ 
these methods is described below, followed by a discussion of its positive and 

negative aspects and guidelines' f or when and how it sho uld, be use d. .In the^ 

cases of questionnaires, interviews, and observations, sample forms for imple- 
mentation of each method are included in an dppendix. The forms are intended 
to be generic examples of the application ^guidelines presented, and they are 
~ltot intended to be extracted and used, £nr«* The forms used if5-any~t-ee4back 
collection effort will likely have to be tailored to the needs ~of the particu- 
lar situation. Not all the guidelines and information relevant to each feed- 
— back-melrho d c an b e^es^t-ed-to-t-h-is-han^boolc, — The-ma^o r point s are rnvpre d 
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here and relevant references for several of the methods are listed in Appendix 
D . Training developers and evaluators planning to make extensive use of a par- 
ti^ilar norhnA should read the relevant references, In addition to this hgnd-_ 
book. Since no method is totally -sufficient in and of itself, the integration 
of methods into a total" feedback system is discussed in the section following 
this one. 

y 

Informal Feedback 

Informal feedback is that which is unsolicited and flows through unstruc- 
tured means. This includes, JafWfBttiffi. .l!«.UL|^M!!M.^£?B!! t ^44» - - 

mouth and; through general written formats, such as letters and completed DA 
Forms 2028 recommending changes in training support materials. Numerous exam- 
ples of feedback fall under this category. To provide external feedback, 
training managers in the field may telephonically or in writing contact per- 
sonnel at the appropriate Center/School about a problem with training support 
materials or the training 6f soldiers recently arriving in field units. In , 
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.some cases, special "hot lines" to the Center/School are. set up for this pur- 
pose* Or field unit personnel may contact training developers about perceived 
problems while they are visiting the Center/School to attend, a conference or 
for some other purpose. Almost all training support materials include a re- 
quest for comments to be sent to the developer, either on a DA Form 2028 or on 
a questionnaire form provided with the_publication._ Requests for general in- 
formation to be sent using DA Forms 2028 are included here as examples of in- 
formal feedback. Structured forms provided with publications for obtaining 
feedback are addressed in the later discussion of questionnaires. 

Internal feedback may also be transmitted informally. For example* a 1 
trainer in an instructional department may informally, contact personnel in the 
DTD with comments on the Program of Instruction (POI) developed for a particu- 
lar course* Or subject matter experts within the. Center/School may informally 
review and comment upon training programs falling within their area of exper- 
tise . The turnover of military personnel within DTD T s provides another means 
of obtaining informal feedback. These individuals may recently have been 
transferred from a field unit assignment , and thus be in a position to comment 
upon the usefulness or likelihood of acceptance of particular materials in the 
field. .< 

Another characteristic of informal feedback which should be mentioned here 
is that it is usually based upon unstructured personal observations. These may... --^ 
be the observations of training managers in the field during ^the conduct ±6~ir" 
training, or the observations of benter/School personnel visiting field tinits 
and. observing training without applying structured feedback collection method- 
ologies. For example, general officers from the Center/School may observe 
"tT^ra^g~whiie ^v1j^^"fS^'f"±ei'3"~ST"its and provide comments in memoranda to ihsti- 
tutional agencies or letters to the field. Internal feedback may be gathered 
through informal observation of the initial administration of a new training 
.urogram, or thro ugh gathering o bservation s from institutional trainers . Infor- 
mal feedback is thus a^y^product of the direct, personal experiences of per- 
sonrxel with the products of the training development process. 

Positive and Negative Aspects 

The primary positive aspects of informal feedback are that it requires 
relatively few resources to collect, it does not detract from or interfere with 
training, it! is usually based upon direct experiences with the products evalu- 
ated, and its open-ended or unstructured nature may lead to new and unexpected 
information . Informal feedback may uncover unanticipated problems with products 
that training developers and evaluators would never have thought to address, 
otherwise. These advantages are significant enough to support the continued 
use of informal feedback* but several potential disadvantages should he^ Tcept in 
mind* : j 

One negative aspect of this type of feedback is that it \ is not as Readily 
available as one might expect. Probably due to time constraints, training mana- 
gers, in the field do not frequently provide unsolicited feedback to training 
developers and e-tfaluatbrs. The flow of DA Forms 2028 into ffTDs could best be 
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described as a trickle. A feedback system which depends heavily upon informal 
information is thus likely, to suffer from a lack of data; The provision of 
informal feedback depends largely upon the attitudes of trainers; draining 
managers, and commanders and the number of informal contacts they have estab- 
lished at the Centers/Schools. _ Some field personnel may have ac n uaint -_^ 
ances at a Center/School and feel free to contact them regularly, while others 
may be hesitant to surface problems outside their units s; The informal feed- 
back available may thus over-represent the observations of particular units 
and individuals and hot provide ah Accurate picture of the overall situation 
in the field (i.e. , the squeaky wheel gets the grease) . No structured sampling 
techniques are applied in gathering informal feedback, so there is no guarantee 
that the samples obtained are representative Ones. 

Another problem with informal feedback is that it tends to be general in 
nature and does not identify specific tasks or parts of training support ma- 
terials where problems lie. It is thus usually necessary to follow up this 
type of feedback with methods which yield more specif ic information. There is 
also usually no guarantee that informal feedback is accurate . It is largely 
based upon personal observations, and these may be over-generalized and influ- 
enced by interpretations and opinions. This type of feedback should thus hot 
be accepted as being definitive, but rather as initial indications of potential 
problem areas that need to be addressed further. A final problem with informal 
feedback is that its lack of structure makes the data obtained difficult to 
tabulate, analyze, arid integrate. The maintenance of the .data may_ thus be 
cumbersome, since they cannot easily be fit into automated data files. 

Guidelines f or— Use 

In spite of. the problems discussed above * informal feedback should be d 
part of the evaluation process. Informal communications should be encouraged 
among all participants in the training development process, since useful infor- 
mation may be transmitted in this way. But informal feedback should not be 
relied upon to provide definitive information to support final decisions about 
revisions needed in training development products. Guidelines for use of inform- 
al feedback in- the overall evaluation process are presented below. 

i; Since informal feedback tends to be general, personnel receiving it 
should gather all the details they can at the time of input . If a problem with 
a product is noted, then all the details of the situation in which that prob- 
lem occurred should be identified. If \a problem with training support mater- 
ials is noted, then the specific parts bf the materials needing revision should 
be identified. * The number of personnel noting the problem should also be iden- 
tified, along with general demographic data indicating their backgroiind/ and 
experience level. These details should be recorded in a written forrn^ and 
filed in an accessible -manner, - Trusting .informal feedback to. memory mjiy lead 
to distortions in the information obtained. : ' ' 

2 While informal, feedback is by definition unsolicited, it car/ and should 
be solicited in general ways. The setting up of "hot lines" in Centers/ Schools 
for the provision of informal feedback should be encouraged, and the telephone 
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numbers of training developers arid evaluatdrs should be made widely available 
to field unit personnel. eommarid letters arid other copnunicatibns distributed 
by Centers/Schools should encourage the return of feedback. Personnel attend- 
ing conferences or short courses at a Center/School should be encouraged to 
contact training developers arid evaluators with feedback on the Use of train- 
ing products; 

3. "When informal feedback is obtained, it should be responded to, whether 
or not it led to a revision in a training product, Those providing feedback 
should be told what was done about problems they identified or recommendations 
they offered. If nothing was done, they should be told why. In this way, per- 
sonnel can become confident that feedback they provide will be attended to and 
that appropriate actions will be taken. 

4. Informal feedback should be used only as an initial indication of 

where problems may lie. It should always be followed up by collection of fur- 
ther feedback using methods discussed below, in order to determine its scope 
and accuracy. Revisions in training products should not be made based upon 
feedback that may be the opinion of one person. Informal feedback can be an 
important starting point in the feedback process, but it is. not the end point. 

Existing Unit Performance Records 

Unit personnel maintain numerous records of the results of the implementa- 
tion of training programs in their units. These records are used to manage 
training (e.g., to determine in what areas further training is needed) and to 
determine the combat readiness of individuals and units. A few of the numerous 
examples of such records are described below, to serve as a basis for discus- 
sion of their usefulness as feedback to Center/School personnel. 

In field units, written reports of the results of the unit's performance 
on recent formal ARTEP exercises are usually maintained at battalion or squad- 
ron level. The results usually consist of an overall rating, narrative descrip- 
tions of gerieral^erforraance strengths and weaknesses, and criterion-referenced 
ratings of the performance of each company-sized unit on various missions. 
Records of individual task performance are generally maintained in the form of 
results of recent Skill Qualification Tests (SQT's). Battalion-level summaries 
of these results indicate the percentages of soldiers Who correctly answered 
each question on the written portion of the test and who properly performed each 
task on the hands-on portion. The results of various qualification exercises 
are generally maintained at battalion or squadron level, and they span the 
individual/collective task dimension. These results may provide indicators of 
training effectiveness at the level of Individuals (e.g., firing of the ^0.6 
rifle), crew (e.g., firing of tank Weapons on Table VIII), and platoons (e.g., 
control of firing arid iovement of tanks on Table IX) . The indicators usually 
consist- of numbers or percentages of individuals or units qualified, and perhaps 
general reasons for non-qualification. Other records indicating the effective- 
ness of training may be maintained at various levels in field units. For exam- 
ple, job books maintained by supervisors are designed to provide a record of 
individual performance on specific tasks, along with narrative comments relating 
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to training needs or problems. Such records are usually not consolidated at 
battalion level. 



Records are also maintained on institutional training which provide meas- 
ures of the effectiveness of implemented training programs. Results of written 
and hands-on performance tests given during or at the end of a course of in- 
struction indicate the proficiencies of soldiers on specific tasks. Records of 
the numbers of soldiers failing to complete particular training programs and 
the reasons why provide more general feedback on the effectiveness of training. 
Results of internal inspections conducted by training evaiuators may indicate 
whether lesson plans and other training support -materials are being used as 
they were designed to be and whether other general problems exist in the train- 
ing base . 



Most of the sorts of records described above are designed to meet the 
feedback needs of training managers within units. They provide these personnel 
with important information which can be used to distribute 'training resources 
where they\^re needed. The key question to be addressed below is whether these 

records can also meet the feedback needs of training developers. 

\ . 

Positive and N kegative -Aspects 

A potential advantage in the use of existing unit records as feedback to 
training developers is based upon the assumption that these records are readily 
available. Few resources should be required to gather data. that are already 
available in units, and the provision of such feedback should place little bur- 
den on unit personnel, since they would not be required to generate any data in 
addition to that normally produced. The problem is that. this assumption is 
often not valid; unit performance records are often not readily available in 
units. For example f ARTEP results may not be available because the unit has 
not part icipated in ARTEP exercises during the past year, or has partic ipated 
in nonstandard ARTEP exercises. SQT results may not be available for all MQS's, 
expecially ? since conduct of a hands-on SQT is now opt ional. Job book entries 
may be incomplete, since supervisors do not have time to record daily observa- 
tions of task performance by all soldiers working for them. 

The unit records that ^re available may not be in a standard format that 
is comparable from unit to unit. For example, ARTEP 1 s and qualification exer- 
cises are conducted differently in different battalions. Range and ammunition 
availability affect the way that major exercises are conducted, such as Table 
VIII in tank battalions. Results of such exercises cannot be compared and 
integrated across battalions without consideration of the conditions under 
which the exercise^ were conducted. Different scenarios and conditions are 
employed in ARTEP exercises in different battalions, maktog JLt difficult to 
combine" results of such even ts""a(^dss units to determine the effectiveness of 
training programs in the field as a whole. Training developers and evaiuators 
should hot assume that unit performance records are readily available iri a form 
that can be integrated across units. The acquisition and analysis of unit per- 
formance records is a more difficult task than might initially be expectod.. 
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Another possible advantage to using existing records as feedback, to 
Centers/Schools lies in the assumption that these records represent objective 
and reliable measures of unit performance. . This assumption is also riot always 
valid. Resource and time limitations in the field often do hot allow complete- 
ly objective evaluations of all aspects of unit perf ormance . _ Unit performance 
records are thus based to soie extent upon the subjective judgments of command- 
ers and training managers serving as evaluators.. For example * ARTEP 1 ! and 
other collective exercises are complex; integrated events, and evaluation of 
performance on them is to some extent subjective. Control of fire arid move*- 
Sent techniques are examples of collective tasks which are usually evaluated 
subjectively. Unit performance records reflect direct experiences with the :. 
implementation of training programs, but real-world limitations frequently 
prevent these reccrds from] reflecting such, experierices in the objective detail 
needed by training developers. Such detail can be increased by providing 
e valuators with extensive training in evaluation and the feedback of informa- 
tion, and by using performance records based upon data gathered using relative- 
ly objective methods, such as the Multiple Integrated Laser Engagement System 
(MILES) or other instrumentation available at the National Training Center 
(NTC) . But resource constraints often limit the training that can be givefr to 
evaluators and the availabil ity of MILES or NTC, so the utility of exist in gV 
performance records on feedback to training developers is frequently limited^ 

A major disadvantage with the use of unit performance records as feedback 
to Center/School personnel is the fact that these records are often general 
rather than task-specif ic in nature. For example, results of ARTEP exercises 
usually delineate major strengths and weaknesses of participatini units* but 
they do not point out specific problems with training programs or specific tasks 
for which further training /is needed. Results of qualification exercises indi- 
cate the percentage 6-r number of individuals or units qualified, but may not 
delineate specific problems that occurred. General records of the results of 
exercises may be sufficient to meet the evaluation needs of unit personnel, 
since these records are supplemented by first-hand experiences gained during 
exercise conduct . That is, commandeers and training managers may be able to 
identify specific areas or tasks for which further training is needed, based 
upon their observations and personal interaction with exercise evaluators. But 
these details are often not recorded in written form, and thus are riot readily 
available as feedback to training - developers . For example, during the, conduct 
of an ARTEP exercise problems may be noted with the ARTEP document itself , but 
such observations axe usually not recorded in written reports of ARTEP ; results . 
General performance records must thus be supplemented with first-hand observa- 
tions or with interviews of experienced personnel. When unit performance re- 
sults are recorded in task-specific detail, they Wre often highly perishable 
and are not. available for a long period of time. \ Memories of individuals par- 
ticipating in exercises are also highly perishable- and beqpme general and less 
accurate over time. The gathering of feedback pasted upon the performance of 
unit exercises should thus be coordinated with unit training schedule s so that 
it can be accomplished during or soon after exercise completion. / 

There are some individual performance records available which are suffi- 
ciently detailed to allow the identification of specific changes needed in 
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training programs or in training support materials. l'^T results have in the 
past provided measures of soldiers' performance oh specific tasks. Training 
developers cblild use such results to identify specific tasks for which revis- 
ions in training are needed. However, the requirement for a regular , formal 
administration of the SQT has recently been eliminated, so these results are 
expected to be less available in the future. Results of tests administered 
during and at the completion of institutional training programs generally pro- 
vide task-specific measures of soldiers' performance . Such results should be 
readily available to training developers for use in redesign of training pro- 
grams. Measures of performance are generally more formal and more specific in 
institutional; than in unit training programs, so existing performance records 
are likely to be more useful as internal rather than as external feedback. 
Within the institution, training developers should be able to check the accur- 
acy of performance records and obtain needed further detail by visiting train- 
ing and testing sites. 

A final concern with the use of existing performance records as feedback 
to training developers is related to the purposes for which evaluation is con- 
ducted. Training developers and evaluators in Centers/Schools gather feedback 
for somewhat different purposes than do commanders and training managers in 
units. Center/School personnel need feedback to determine the effectiveness of 
training programs and training support materials ~ 9 and to identify any revisions 
needed. They thus have no need to identify specific units or individuals from 
whom feedback is collected. Field an it personnel need feedback to determine 
the readiness of specif ic in^ivi^ further-strain- 
ing needed. Existing unit performance records are designed primarily to meet 
the evaluation needs of field unit personnel . These records are thus largely 
informal , highly perishable, and supplemented by first-hand experiences rather 
than by written detail. They often do not meet the evaluation needs of train- 
ing developers, and redesign of them to meet these heeds might reduce their use- 
fulness to unit personnel. / For example, knowledge that their performance rec- 
ords were to be provided as feedback to a Center/School might influence unit 
personnel to "scrub" such records to make certain that they would not reflect 
negatively on the unit . The obj ectivity of exist ing unit performance records 
can- be questioned, and the provision of such records to. higher headquarters 
migl|t produce further problems here. Also, unit personnel do not have t ime to 
record and maintain performance data in the detail that they are needed by 
training developers. The management of such data might "reduce the amoun t of 
time \ that senior unit personnel coulcl spend in f irst-hanci monitoring of training 
activities. Training could become a set of formal, discrete activities to meet 
the heeds of Centers/Schools^ rather than a continuous adaptive process to meet 
the needs of units. It is thus best that unit' performance records remain 
oriented toward meeting the evaluation needs of unit personnel, "and not be tai ^~ 
bred t6 meet the heeds of Center/School personnel. These records can provide 
some usWul information to training developers_, but they cannot in and of them- 
selves meet all of these personnel T s feedback needs. / 

As was suggested earlier, records based upon the conduct of/ exercises 
using MILfes or instrumentation available at NTG may provide feedback which 
alleviates some of the problems discussed above . Such exercises generally 
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provide results which aire relatively objective. and detailed,, so these data 
should be gathered as feedback whenever possible. However* limits oh the 
availability of MILES, equipment and restrictions on :he use of NTC data may 
limit the extent to which results of/ tactical engagement simulation can be 
used as feedback. The use of simulations and automated data collection devices 
provides a promising approach to the efficient ga the ting of objective feedback 
in the future. But until such simulations, and devices are routinely available, 
caution should continue to be applied in the use of existing records as feed- 
back. 

V 

Guidelines for Use 

As discussed above, there/are many problems with the use of existing unit 
performance records as feedback to Center/School personnel. These records are 
often not readily available , /not available in a . standard form, not totally ob- 
jective, arid riot available iA sufficient detail. But since they do reflect the 
results of the impl emen ta t ion of training programs in at least a general sense, 
unit performance records can have some utility in a total feedback system. 
These records should be u£ed as feedback in acc6rdance with the guidelines pre- 
sented below. 

1. Since existing/unit performance records are often not available in suf- 
ficient detail to meet /the needs of brainfag developers and evaluators, as much 
detail as can be used /should be gathered to supplement these records. If the 
results of —a-qual-if icgtion or-ARTEP-exercise are collected, -details on how the 
exercise was conducted^ and scored should be gathered, training developers and 
evaluators can accomplish this by interviewing exercise participants or by 
observing the conduct of the exercise. Since detailed- performance records arid 
individuals' memories are 7 highly perishable, supplemental information such as 

scenarios used, problems with scoring or targets , and. specif ic_ tasks_.or,_sulvr_ 

tasks resulting in performance difficulties should be gathered as quickly as 
possible following exercise completion. Performance results should be gathered 
in as original a form as possible; e.g., original scoresheets should be gathered 

in addition to summary results . .. -iflien_o.bservin.g. the cfonduct—o£ _an_- exarcise^ 

training developers and evaluators should spotcheck the accuracy of scoring 
decisions and record factors which may influence the results obtained. Addi- 
tional details should be obtained in as direct and tAggly a manner as.Js^po3Si" 
ble, and plans should be developed for use of these data prior to their collec- 
tion . 

v 

2. When ga t her irig unit performance records as feedback, the anonymity of 
the units and individuals^ in volved should be insured. Identifiers of individ- 
uals and units should be removed from scoresheets and from all other data col- 
lect ion forms . Assurances of anonymity should be given to personnel in units 
from which performance results are collected. In this way, tendencies to bias • 
results or cover up problems cari be reduced . N The feedback gathered should not 
be used in any way which might reflect negatively upon particular individuals 

or units. 




3. Performance results are generally gathered more formally and iri more 
detail in the irist itut ibrisii setting than in field units. The. gathering of 
supplemental details is also less costly in the institution than in the field. 
Existing performance* records should thus be more heavily relied upon as inter- 
nal rather than as external feedback. Training developers and evaluators 

should obtain results of performance tests and exercises from training programs 
conducted within their Center/School. They should also visit institutional 
training and testing "sites in order to gather further details arid to check the } 
accuracy of the data obtained ,• \ 

A , As is the case with informal feedback, existing performance records 
should be used as initial indicators of where problems may lie iri the training 
process, and not as final, definitive 'feedback. These records should be sup- 
plemented by interviews, observations, arid other feedback methods, in order to 
insure the. accuracy of the feedback obtained. Existing records should not be 
heavily relied upon in a feedback system, but should be used as an indicatipn 
of areas in which further information is needed'. 

Quest ibririaires^ / 

Perhaps the quickest way to obtain information for feedback purposes /-from 
a group of respondents is to require them to complete a questionnaire, A ques- 
tionnaire is any ordered set of quest ions or items des igned to _js]y^_^^j^teg — 

responses from a specif ied pop ulat ion or -group . Broadly- speaking, -quest ion- — 

naires may take on a variety of forms ranging from rating scales to multiple 
choice items to open-ended questions. Open-ended questions are ones that. allow 
the respondent to answer in his/her own words, including whatever qualifiers 
he/she likes. Some forms lend themselves better to collecting accurate feed- 
back- than- others— ^ — Xhexe_ 

are a number of other factors influencing the accuracy of information obtained : 

through questionnaires, which will also be discussed later iri this section. 

Although questionnaires can be used for gathering feedback in a Variety of 
situations, the power of the questionnaire is best demonstrated when you. have a 
large group to survey and few data collectors, or little time to gather the 
^a t a r 

taining feedback, and possibly the-only method . Questionnaires can be adminis- 
tered to groups or individuals in the /field or at the institution for obtaining 
quick' feedback on personnel or train trig matters. Questionnaires are flexible / 
iri that they can be administered in person or through the mail. However, mail-/ 
irig out questionn aires is-^ iatL recommended jor _reasons to be discussed later. './ 

Positive and Negative Aspects / j _ 

As mentioned above the questionnaire can be used to gather a considerable 
amount of feedback in a relatively short time with only a small investment ija 
resources. A single data collector armed with multiple copies of a question- 
naire can collect information f/om several hundred respondents in a relatively 
short period of' time. If the Questionnaire has been carefully designed, the 
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same data collector could ^ with the aid of computerized data management i 
analyze the data and provide feedback on the results in a matter of days. 

In general, research has shown questionnaires to be a reliable way of _ 
-gjatlTe^Tg^^ amblgously or beyond 

the reading level of the respondent, questionnaire responses tend to be con- 
sistent with the intent of the questions, the consistency in answering ques- 
tionnaires may in part be explained by the fact that each respondent reads the 
same questions, worded in the same way, in the same order on each occasion- 
This allows the collector of feedback to combine the responses of the Individ- 
uai Respondents for arriving at general conclusions concerning the development 
and implementation of training products. 

Questionnaires allow the training. developer or evaluatdr to gather infor- 
mation that might be extremely difficult to see or might take months to obtain 
through direct observation. In essence, the questionnaire expands one's abili- 
ty to. observe by letting the respondent act as the eyes and ears of the would- 
be observer. While this expanded ability to observe makes the questionnaire 
extremely efficient, it does hot guarantee that responses to the questions will 
be accurate. Accuracy may be adversely affected by the Inability of the re- 
spondent to recall relevant information or by the respondents unwillingness to 
answer questions completely and honestly. Honesty and memory issues will be 
discussed further in the guidelines for using questionnaires at the end of this 
section. _ * ' ; 

The biggest drawback to questionnaires is that bad questions are easy to 
write. Too often questionnaires are written for no reason other than to meet 
an evaluation requirement . Often those writing the questionnaire halve little 
idea of what information is needed or what they will do with the information 
once they obtain it. The result is the proliferation of a large number of 

' questionnaires ™" r a i n in p^y^aieral-q^es^^-JLba^ p r oduc e data that „are no t 
useful for improving training programs or products. Such Questions as What is 
the present state of training in your battalion?" or "How well trained are sol- 
diers when they arrive in your unit?" are unlikely to provide information about 
the extent and effectiveness of the training conducted in the battalion or about 
the tasks the soldiers could or could not perform upon arrival in the unit. To 
retain the, latter information, more specific questions are need ed , perhaps sup- 

^pTemented "by observation and testing. 

Lack of specificity in questions reduces the. amount of useful information 
that they produce. To make matters worse many questions call for subjective 
responses, consisting largely of the respondent's impressions, opinions, or 
subjective evaluations of training programs, materials, or soldier proficiency. 
Responses are typically obtained in terms of five or six-point rating scales, 
which have been used in a wide variety of applications, often inappropriately. 
Some of the more common uses have been for obtaining task criticality ratings, 
relative task difficulty ratings, and estimates of soldier proficiency by super- 
visory personnel. Bating scales have also been used to evaluate training effec- 
tiveness, training materials, and in numerous other applications where there are 
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more accurate. ways of obtaining the information required.. Because rating 
scales are relatively easy to design and quick to administer and score * they 
tend to be overused-. Even when used appropriately, -the accuricy--6JE--inf brmatibri- 

obtained via rating scales is" adversely- af fected by rater -^TTOM-f- rAmonj^h^ 

most common rater errors are rater leniency , central texidency errdts ] arid~ the 
halo effect. Leniency errors occur when raters avoid using the low extremes, 
of a rating scale. These errors often arise because raters do hot wish to give 
ratings that can be interpreted negatively. Other raters may be biased against 
giving extreme ratings, either high or low. When both high and low extremes 
are avoided* ratings tend to cluster around the mid-point of the scale. When 
ratings cluster around the mid-point, the differences among things rate d may be 
so snail as to be of little practical significance. The halo effect occurs 
when the rater bases all ratings on a global or overall judgment of the person 
or thing being rated. For example, a supervisor asked to rate a soldier s per- 
formance on specific tasks may rate him or her_high on all tasks based on the 
global judgment that the soldier is a good worker. The supervisor may give 
equally high ratings on all tasks even though some of the tasks were never ob- 
served and the soldier does not perform all tasks equally well. This rating 
tendency detracts from the ability to c discriminate between different aspects 
of performance. 



The accuracy and usefulness of in f o rma t ion ob t a ined by „ qu^s c ionn a ire s 

largely depe*ds„on the questions asked and how they are asked. Often the word- 
ing of a question may indicate" the expected or preferred -response .- Such-ques^... 
tions are referred to as leading questions. Two examples of leading questions 
are given below. 

1. which tank do you prefer, the faster, lower profile Ml or the larger, 
slower M60? ^— ' 



2. Do you feel the" weapon systems are not reliable, and, vaore trequently 
than not, do not operate to the full potential for which they were designed? 
Such questions tend to support the preconceived notions of the questionnaire 
developer rather than to obtain accurate feedback. 

Anc&her-problea-w-it-h-questionnaiTPs i **,that _the-y_ are__unpo^ular, with^tbogg _ 

persons from whom the feedback must be obtained. The unpopularity of question- 
naires increases as the number and length of questionnaires to be completed xn- 
creases- due to fatigue -and -other factors'. ' Txeld personnel perceive, often 
correctly Hiit the questions asked, have n'o real purpose and that no meaningful 
cnanges will be made^s the result of their responses. Thus they do not devote- 
as much thought to the questionnaires as/they could. 



Guidelines for Use - ■! 



Because they provide an eff icient means of obtaining information, ^stlon 
naires should be considered whenever there is a need for timely feedback f «p* 
the field and resources are limited. Although no, feedback effort should rely on 
Questionnaires alone, well-designed and carefully administered 
can provide a substantial amount of useful information. To help in the design 
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and administration of questionnaires, the following guidelines are offered, 
Following these guidelines should improve the usefulness and accuracy of the 
information obtained. _Twb sample quest ionna ires " exempl ify lug many- t>f- -these--- 
guidelines are included at Appendix A., 



1. No questions should be written until the questionnaire developer iden- 
tifies what information is needed and how it will be used. When it has been 
determined what information is needed and. how it is to x be used, the next step 
is to determine if questions can be formulated to obtain the desired informa- 
tion. Given a question that is likely to yield accurate information, the 
feedback collector must next identify the person or group who is in the best 
position to provide this information. Factors to be considered in selecting 
those who will provide the information include whether they might have direct 
knowledge or recent experience pertaining to the item and the likelihood that 
they will answer the question openly and honestly. To encourage openness and 
honest-, the person responding to the questionnaire and his/her unit should not 
be identified whenever possible. If the respondent's anonymity cannot be. in- 
sured, then additional information supporting the accuracy of the questionnaire 
responses may have to be obtained. If the questionnaire asks for information 
of a sensitive nature and responses aire not anonymous, then Privacy Act co nsia- 
era t ions come into play. 



2. Questions must be worded so that they are direct and ..to the point 
T ov shou] d be readily understood by the respondents for whom they are .designed. 
The questionnaire writer must be careful to use terms that mean the same to^each 
respondent as they do to the Writer. Ambiguous questions that depend heavily 
for an answer on the respondent's own frame of reference should be reworded or y 
omitted. For example, the question "Are crews selected ^ fly together ,os f' 
the time'" is ambiguous and may be answeredyes-jDjr_^a-^ependingon--t-he respond- 
ed '^^eT^ret^i^c^e^ras^most- of-the'Time^ The question -ght bet- 
ter be worded in terms of how often or what percentage of the time crews fly 
together. Questions inquiring about the adequacy, sufficiency or satisfactori- 
ness of this, that, or the other tend to be ambiguous, as do questions asking 
whether certain events occur frequently, occasionally, sometimes, regularly or 
often. ; 
3, To -be-really -useful feedback must be^specific ^d^his requires the_ 

~-use-et-spec-i^^^ 

"wK is your opinion of", "what do you think of", and similar phrases are un- 
liSlv to produce very much useful feedback. Responses to such questions will 
vary widely from one respondent to the next , making it difficult to draw any 
conclusions from the respondents as a group. Furthermore, questions phrased 
this way are more likely to elicit general opinions than to generate specific 
factual information. To obtain specific factual information, questions must 
precisely sp^ify what information is required. Instead of « -tog: . t«k 
ner how he feels about the unit's gunnery training, you might ask him what live 

\ fire gunnery exercises he has participated in since arriving in the unit. Rath 
er thfHsk the platoon leader what he thinks of a particular A^TEP document, 
you might ask hiS/her which parts of it were ^ed d uri ng the_most recent^RTEP, 
and then ask what problems were encountered in using those parts of the ARTEP , 
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, S. When open-ended questions are used, the questions should be sharply 

focused. Questions such as\ '-'What equipffien-t--do--you-presenX±y : -u : 

shooting the ML turre t?" an d "What equipment do you need for turret trouble- 

shooting that you. don't now have?lL.are examples of sharply focused questions: 

These questions are preferable to general open-ended questions, such as "What 

do you think of the equipment you have for troubleshooting the KL turret?" 

Whenever possible, open-ended questions should be replaced by questions offer- 
ing specific response alternatives. In the example above, the questionnaire 
developer may know in advance what equipment the mechanic has available for 
troubleshooting the Ml turret. Therefore, he/she can list the available 
equipment (e.g.> STE ML, multimeter) as response alternatives and allow the 
respondent to check those that he/she uses in troubleshooting the turret. 
Providing response alternatives saves time in administering and scoring ques- 
tionnaires- and is highly recommended. -Sometimes in pretesting a questionnaire 
on a small number of "respondents the range of response alternatives to some • 
open-ended questions will become apparent. When this occurs the response 
alternatives to these questions can be supplied in the final version of the 

questionnaire. Not every open-ended question, however, may be changed in 

this way since a given question may have too many po s s ibl e - r e spon se--alt erna t ives— 
to list . , 

5. Certain types of questions should be avoided because they do not yield 
accurate, use.' 1 data. Other types of questions should only be used under very 
special circumstances and on a limited basis. Leading questions, in which the 
wording of the question"" implies an expected or preferred response (as. in. 'the 
examples given earlier) , should be avoided. Also avoid phrasing questions m 
negative terms. Respondents often overlook the negatives in reading such . 
questions and interpret the questions incorrectly. Negative questions are 
often leading quest ions as was illustrated in. the example mentioned earlier. 
Do not ask two different questions as one compound question. For example, the 
question "Was the feedback you received complete and accurate? actually c °n- ... 
sists of two separate questions (i.e., was the feedback complete? and was the 
feedback accurate?). Such compound questions should be avoided because a person 
can agree with one part of the question and disagree with the other. Rating 
scales and other subjective appraisal techniques should be used cautiously in 
questionnaires 4esi^ed_tq_provide £eegback.. Rating scalea .seldom provide tfee .. 

kinds of 'de"r.a iled information that are required in making specific changes to . 

training programs and materials. If 'It is determined- that a rating scale can 
provide the desired information, then appropriate guidelines offered in the 
references in the "Questionnaire" section pf Appendix D should be followed. In 
addition, if rating scales or other subjective appraisal techniques are used, 
the accuracy of these techniques should be checked by comparing them with ob- 
servational or performance data prior to. using tjiem to collect feedback. The 
only exception to this rule is when the data of interest are the appraisals, 
judgments, or perceptions themselves (e.g., judgments concerning |ask critical- 
ly br measure! of job satisfaction). Generally subjective appraisal techniques 
should not .be used for determining soldier proficiency or effectiveness of, 
training, programs and materials unless it can be clearly demonstrated that the 
sub j eSfvfmfaSres accurately reflect proficiency and effectiveness- as -measured. 

by jmxe~.bbi^ 



6.. If rating scales are to be used* make the scales as explicit as possi- 
ble. Phrase the scales in terms of explicit observable measures of performance* 
rather than in vague general terms such as "average 11 , fl belbw average", etc. 
Describe each rating point in terms of the behavior that it represents. Con- 
sider asking raters tp provide specific examples to support their ratings. . Be 
s u r 6f~t hat 'the rat ers" ha ve t Ke~ exp erienc e ~6 f bac kg round to r a t e~ wha t i s ~ b e in g 
rated. Give raters the option of indicating that they have not had experience 
relevant to the items being rated. Train raters in making subjective appraisals. 
Rater training should include experience with the rating scales to be used; a 
discussion af^common errors (leniency arid halo effects) associated with ratings, 
arid a discussion of the dimensions of the situation being evaluated. 

7. Questionnaire items should be obvious and straightforward. There 
should be no need for the respondent to stop andfigure out what the question 
means or to search his/her memory for long-forgo trt en facts. Questions should 
deal primarily with events that have occurred -i n ' the^last- fewdays or weeks, or 
with easily remembered events. In forming questions which depend on a respond- 
ent's memory or recall capabilities, the time period covered by the question 
mus* be carefully defined; the "when" should be specifically provided. Rather 
than ask the respondent what training he/she has received recently, ask how many 
hours of MOS training he/she received in the last three days or last two weeks. 
Do not expect to get accurate, detailed accounts of events that occurred many 
months or years ago. Questions can/address events occurring months or even 
years ago when the event is likely to be especially significant or salient to 
the respondent, and absolute accuracy and fine details are hot required. 

8. Questionnaire it ems should be arranged in a logical sequence, both to 
facilitate the respondent's recall of the information and to shorten the time 
needed to answer the questions. In general, questionnaires should require no 
'nrore~thBn~air~ : ho Q r o f t:he-rrespon-dent :t -s — time for complet ion. Shorter question- 
naires are recommended. Do not. pad the quest ionnaire with add it iona 1 it ems just 
to make it longer. Do not include questions that elicit information that is 
only -o f-m i-1 d-in-t e rest - — Quest ions _ JJicl_ud_e d . s ho ul d _ _b e t ho s e that produce data 
having a known use. Unless a use for the data produced by a question can be 
specified in advance, the quest ion should probably be discarded. 

9. The questionnaire should have a title or heading to* identify it on each 
page. If the questionnaire consists of more than two pages, the pages should be 
numbered. Each- JLtem. JLric luded in the questionnaire should also be numbered,. Be 

-s u re- to— 1 ea ve- enough— room -under-ea ch-question -for—the -respondent— to--wr-ite-^an-— — 
answer to the question. While the order of the quest ions is not critical , a 
logical arrangement of items by subject matter or chronological order facili- 
tates questionnaire administration . Quest ionnaires should usually include writ- 
ten directions for completing the quest ionnaire. Directions should always be 
clear and prominently displayed. Generally, respondents should be told' the pur- 
pose of the quest ionnaire either verbally by the test administrator or as part of 
the written instructions. 

10. Distributing and receiving questionnaires through the mail is not recom- 
mended . The return rate of ma iled-out ♦ quest ionnaires is typically so low that 

19 



^7 



little confidence can be placed in conclusions drawn from the select few that 
are returned. Much less thought and effort is usually devoted to a mailed 
que s t ionn aire than would be devoted to an interview. Wh :Qe this statement 
p_rpbably holds_t rue PJLS 1 *?. 3 ? ionnaires in general, questionnaires received in 
th£ mail probably receive even less attention "Fhan other quest ^ionnaires. There 
is much to be said for administering questionnaires in person. Not only do you 
get more completed questionnaires, but the effort invested by respondents in 
completing the quest ionnaires is generally greater. Also the administrator is 
available for handling any questions that arise about the questionnaire items 
and for checking each questionnaire for completeness as it is returned. When 
a questionnaire is- administered to a group, all respondents should be kept at 
the site where the questionnaire is administered for a predetermined period of 
time or until all respondents have completed the questionnaire. Respondents 
should not be permitted to leave the site early. 

11. All newly developed questionnaires should be pretested prior to using 
them to gather feedback. The quest ionnaire should be pretested on the same 
type of respondents as those for whom the questionnaire was developed. For 
example, if the questionnaire was designed for tank commanders in an Ml battalion, 
then it shou^ be pretested using Ml tank commanders. In pretesting the question- 
naire, have tfhe respondents read each question, explaining what it means in their 
own words. Then ask the respondents to answer the questions. By noting instances 
in which questions are not understood or misinterpreted arid which questions pro- 
duce little information, the questionnaire can be refined. Such pretesting may 
result in rewording some questions, focusing others more specifically and elimi- 
nating still others. Pretesting may also suggest additional questions to be 
included in the final version of the questionnaire . 

Structured Interviews 



Another efficient method of obtaining information for feedback purposes is 
through the use of structured interviews. A structured interview consists of a 
prepared set of questions verbally asked, on a one-to-one basis, of one or more 
interviewees by a skilled interviewer . Typically, the same questions are asked 
in the same order arid in the same Way each time the interview is conducted. 

Structured in tervj : ews_usually work be st ^when the infor mation gatherer^gs a 
clear idea of the "kind "of information he/she is seeking. For training developers 
looking for specific information about particular products, the method is there- 
fore ideal. Interviews tnay be conducted at the institution with instructors or 
with soldiers just prior to their graduation from any given training program. 
Interviews may also be conducted with soldiers coming back from the field to 
attend a resident course at the institution.' Soldiers returning to the insti- 
tution for training are an especially good source of information for training 
developers who tnay otherwise iiave little contact with personnel in i the field. 
During BTT visits, graining developers and evaluators can interview unit person- 
nel to assess the state of training in the unit and to evaluate the products of_ 
the training development process. The information thus obtained can be used both 
by unit leaders to better manage unit training and by training developers at the 
institution to improve their training products. 
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Positive and Negative Aspects 

Although structured interviews require more time arid effort than question- 
naires, the payoff in terms of quantity and quality of information collected 
from interviews is greater. Unlike, questionnaires > structured interviews do 
not require the respondent to give his/her answers in writing. This becomes 
more of an advantage as the motivation to write and writing skills of those 
being queried decreases. Thile some NCO's or enlisted personnel may be per- 
fectly willing and able to answer a question in great detail during an inter- 
view, providing the same answer in writing may be another matter entirely. 
Thus, the ; interview is likely to provide more detail about each item of infor- 
mation sought . 

Like the questionnaire! the structured interview insures that the same 
questions are asked each time the interview is conducted. This enables the 
recipient of the feedback to combine or tabulate responses, for each item of 
interest. If the interview were not structured, as is characteristic of many 
interviews presently used for feedback, the tabulation and combination of re- 
sponses would be diffip**fT^ Having each interviewee answer the same questions 
and combining the interviewees' responses to the questions allows the inter- 
viewer to determine Aether a problem is widespread or merely applies to one or 
two persons . 

Structured interviews allow the collector of feedback to gather informa- 
tion on events that he/she was not personally present to observe. As a result 
it tends to be a relatively efficient method of obtaining information on events 
that were not or could-not be directly observed by the feedback collector. [ 

While the structured interview has the questionnaire's advantage of pro- 
viding the same kinds of information for each interviewee, it has flexibility 
that the questionnaire does not possess. During ah interview the interviewee 
may indicate through his/her answer or in some other way that he/she does not 
understand the question being asked. When this occurs the interviewer can re- 
phrase the question or otherwise clarify what is being asked. If the -answer 
given by the interviewee is incomplete or raises additional questions, the in- 
terviewer can probe for further information. In this way, the interview can 
lead to unanticipated and perhaps more informative responses - 

Interviews - "are generally better received by those who are being asked to 
furnish information than are questionnaires arid other kinds of surveys. While 
this may seem to be a minor advantage, it is actually an important reason to 
use the interview method. The quantity and quality of the information gathered 
depends, in part, on the willingness of the persons surveyed to provide it. 
Individuals who are reluctant to complete a written survey may . look favorably 
upon being interviewed. They often appreciate the opportunity to express their 
ideas openly to an interested, impartial listener. 

The primary disadvantage of structured interviews as a method of gathering 
information is that the method requires a substantial investment in resources. 
Interviews require committing one or more skilled interviewers to the task over 
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ah extended period of time; Interviews must be done on a brie-tb-brie basis, 
with each interview lasting from 20 minutes to an hour. Because several per- 
sons at each level must typically be interviewed, the interview process can 
consume a considerable amount of time, depending on the size of the unit or 
organization being surveyed. In addition to the time spent .conducting the 
interviews, reviewing and analyzing the information provided during inter- 
views require many hours of a skilled analyst's time. Although data must be 
analyzed, no matter how they are collected, the data collected during inter- 
views are more likely to consist of lengthy narration^ Such narrative re- 
sponses must be read, interpreted, and combined with similar responses for 
drawing conclusions, making the analysis of such data a difficult, and tedious 
process . 



Tile use of structured interviews requires a considerable investment in 
time prior to collecting the data for designing questions and planning the data 
gathering activities. Evaluators must often work hand- in-hand with training 
developers in order to be sure that the appropriate questions are asked in the 
most effective and efficient way. To "design questions and plan the data col- 
lection the evaluator/training developer must decide in advance what informa- 
tion is needed and who is most likely to be able to provide it. When the data N 
gatherer has little or no idea of what he/she is looking for or has little time 
to plan the data gathering activities, the structured interview approach may 
not be the preferred method . In such cases, the data collector may need to 
initially use unstructured interviews or informal observation techniques to 
gather enough information to develop structured interviews for use at a later 
date. -/ 

Another potential problem with using structured interviews /is that they 
require consistency and impartiality on the part of the interviewer. After 
asking the same questions in the same way a number of times, the interviewer 
may tend to become bored with the process and start to rephrase the questions, 
skip questions, press the interviewee for a quick answer, or unnecessarily 
prompt the interviewee in order to speed the interview. Unless the inter- _ 
viewer consistently asks the same questions in the Same way, the quality of the 
information obtained will be adversely affected. Not, only must the interviewer 
be consistent, but he/she must be careful to maintain impartiality throughout 
the interview. If the interviewer shows by his/her reactions that he/she wants 
a particular question answered in a particular way, then some interviewees may 
be so anxious to please the interviewer that they will be influenced to give 
the approved response. Although the interviewer often influences the interviewee 
unwittingly through a smile, nod, or by displaying additional interest an 
^scrupulous interviewer will frequently lead the interviewee to give the de- 
sired answers to each question throughout the course of the interview. 

Guidelines for Use , . / 

Despite some disadvantages associated with structured interviews, the method 
is highly recommended and should be included as a regular part of gathering in- 
formation for feedback. Used correctly, this method can provide a great deal of 
detailed factual information and answer many different kinds of feedback 
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questions. Many of the guidelines listed for questionnaires are also applica- 
ble to structured interviews. Additional guidelines for using structured 
interviews for collecting feedback are presented below. .Examples of inter- 
views developed in accordance with these guidelines are included in Appendix B; 



1. Interviews should be conducted with P ers °ns at different levels of the 
organization. Persons occupying jobs at the lower echelons should be inter- 
viewed as well as those holding leadership positions. Information is best ob- 

. tained from those persons having first-hand knowledge of the situation. . Those 
working at lower levels sometimes have information that is not available to 
their leaders. They may also provide a different perspective of events affect- 
ing the organization. Too often, only those occupying leadership positions are 
interviewed, jarhich tends to present an incomplete and sometimes biased account 
of events . ^ 

2. The questions asked of interviewees should address areas that the in- 
terviewee is likely to have personal knowledge of * or experience with. There- 
fore, different questions may need to be developed for individuals at different 
levels or occupying different jobs. For example^ the trainers in a unit might 
be asked about the training they conducted in the last two weeks, while those 
who are likely to have received training might be asked about the training that 
they received during the same period. Oddly enough the answers given by the 
trainers and the soldiers being trained do not always agree. This alone is 
reason enough to develop questions that can be asked of persons at different 
levels of the organization. Examples of questions asked at two different lev- 
els of a tank company are provided in Appendix B. 

3. Interview questions should be directed toward identifying significant 
events or obtaining factual information . Questions asking soldiers for their /' r 
opinions, attitudes, or evaluations should be kept to a minimum. For example, 
if you are interested in the training the soldiers are receiving in their unit, 
ask about the number of hours of training received in the last two weeks, the 
specific tasks that were trained, and whether soldiers actually got to perform 
the tasks during the training. Do not ask soldiers to rate their unit training 
on a f ive-po int scale or to indicate how they feel about the training they re- 
ceive in this unit. The former questions are specific and factually based; the 
latter are global and based on opinion . Specific questions call for specific 
information, while global questions generate very general and often useless^ in- 
formation. Questions requiring soldiers to compare a large number of unspecified 
alternatives before selecting one or 190 re of them are difficult to answer and 
should be avoided, especially when interview time is limited. For example, 
asking soldiers to identify the hardest tasks that they have to perform or asking 
them to ident if y the most critical tasks that they perform Seldom yields much 
useful information . 

4. Questions must be worded in a way that is meaningful to the soldiers^ 
If terms used in the questions are intended to take on a particular meaning, the 
interviewer must define the terms so that the questions reflect the intended 
meaning. For example, when mechanics are asked how many hours they have spent 
in performing maintenance, their answer may refer to the number of hours they 
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were on duty or to the time they actually spent in troubleshooting or to wrench 
turning. If the interviewer is only interested in wrench turning, then the 
question may need to be phrased or explained in that way* Similarly , if a 
question inquires about the training conducted, the interviewer may need to 
describe the kinds of events that are considered to be training events. 

5. As much as possible interviews should be conducted in a relatively 
private, quiet location where interruptions and other distractions can be kept 
to a minimum. In this way the flow of the interview is hot impaired arid the 
confidentiality of the interviewee's responses is maintained. However, a quiet 
location is riot always available, and the interviewer is sometimes forced to 
conduct interviews at the soldier's work station in the presence of vehicle 
noise and other soldiers. When soldiers other than the interviewee are present , 
they tend to inject their comments into the interview^ sometimes inf luenc ing the 
responses of the interviewee . When this occurs about all the interviewer can do 
is to politely ask the soldiers to refrain from commenting or suggest to those 
interrupting that they will get their chanc e to answer the questions later. 

6. The persons conducting the interview should be intimately familiar with 
the questions to be asked and be prepared to clarify any questions that are not 
clear to the interviewees. On the other hand, the interviewer should stick to 
the script as much as possible, reading the questions as written, and deviating 
from the interview structure only when necessary. The interviewer must not show 
by his actions or words that he/she prefers one kind of answer over another. 
He/she must also avoid leading the interviewee to answer in a particular way. 
The interviewer must also record the interviewee^ s responses just as they are 
given, using the interviewee's words as much as possible. Care mast be taken 
not to interpret the data or draw inferences from responses before they are re- 
corded on the interview form. Sice the data collection is complete, then irif er- * 
ences and conclusions can be drawn from the data. 

7. To facilitate analysis of the information obtained, interview questions 
should be designed so that they elicit responses that can be coded and counted. 
This does not mean, however, that all questions must be an swered by a number or 
a yes/no response. Of course, numerical and other countable responses make 
analysis mach easier and quicker, and should be used when they provide the nec- 
essary information. Countable responses alone, however , may riot proivde enough 
information to allow meaningful changes to be made. Narrative responses pro- 
vide much of the detail and specificity that training developers need to improve 
their products. Caution must be used, however,' riot to place too much confidence 
in the narrative responses of any one interviewee. Typically an interview 
should elic it narrative as well as countable responses and- should not rely en- 
tirely on either type of data. 

Systematic Observation 

An excellent* but seldom used* method of objectively obtaining feedback is 
through systematic observation. Systematic observation consists of methodically 
recording information about events as they occur, utilizing worksheets that list 
the events to be observed. This may be distinguished from casual observation in 
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which the observer is assigned, to observe a particular activity arid write down 
his/her observations or overall evaluation of it. Systematic observation re- 
quires the observer, at a. minimum, to record his/her observations for each of 
the events listed on ah dbservat ion worksheet . Additional comments may be 
included as needed to clarify recorded observations. The casual observer, on 
the other hand, : usually operates under few^ if any , con strain ts and is free to 
record or not record whatever observations he/she wishes. 

I 

Systematic observation. may be used to gather feedback on training programs, 
the use of training _;aater_ials ^ and for observing tests and evaluations 01 sol- 
dier performance. In combination with hands-on tests, observation is the pre- 
ferred method for evaluating training programs. In addition to the guidelines 
presented at the end of this section , several good references providing de- 
tailed guidance in training program evaluation (see Appendix D).are available. 
These references may be very helpful in planning the observations and designing 
the necessary forms. In additon to evaluating formal training programs, obser- 
vation can also be helpful in assessing the state of training or proficiency in 
units, especially as a check on information obtained through questionnaires arid 
interviews. For example, specially designed observation worksheets can be used 
to spot check unit training or other unit activities such as maintenance. Exam- 
ples of worksheets for spot checking training and maintenance activities in a 
tank company are included in Appendix C. To illustrate different worksheet for- 
mats, the training observation worksheet is presented in two different formats. 

Positive and Negative Aspects 

Systematic observation is perhaps the only method of obtaining feedback 
that provides training developers arid evaluators with first-hand information 
about classroom and unit activities. In systematic observation, unbiased ob- 
servers directly observe and record training events or other unit activities as 
they occur. In this way the .information does not pass through the selective 
filters of memory and self interest that may distort information acquired 
through interviews and questionnaires. Thus the information obtained through 
systematic observation is generally more objective and accurate than question- 
nair.e and interview data, assuming that the observers are objective and unbiased. 

In systematic observation, the observer knows in advance what observations 
he/she will make. He/she has been trained to record the data accurately and ob- 
jectively. Because the observer knows what he/she is looking for and records it 
accurately, the chances are improved that the needed information will be ob- 
tained. Many things that might go unnoticed by the casual ° hs f TVer L 
picked up and recorded by the trained observer, particularly when aided by the 
use of structured observation worksheets. Aside from the information recorded 
on the observation worksheets, direct observation of the problems associated with 
various unit activities may suggest the causes of the problems and solutions to 
them. Insight may also be obtained regarding which changes are feasible and 
likely to be most beneficial . 

Systematic observation provides an alternative to more subjective „ 
for evaluating training programs. Observational methods can be used to determine 
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if the trairing is c-r.ducted in accordance with the training plan, and to docu- 
ment the occurrence of unplanned events during training. They can a so e used 
to provide accurate information about the training environment. Tests can be 
observed to insure that the testswe conducted and scored properly and that _ 
test remits are not contaminated by poor test administration or biased scoring. 
Whereas questioning trainees', instructors, and examiners may in ^eory prov.de 
essenTiaUy the same information about the training P-f^^bv direct o^ser- 
validity of the information obtained is increased substantially by direct 
vations by unbiased observers of the activity being evaluated. 

The 'biggest drawback to using systematic observation is that the method can 
be veS timeSonsuming. When used for spot checking unit ^^^^ata 
nance activities for the purpose of verifying interview or ^ 
observational data can be collected in a E^fg* -jS^J^^ such Is 
if systematic observation is the primary method of obtaining feedba , 
mighJ be the case in -aluating a training program the ^J™^*^*** 
more observers may be considerable. The time necessary P contain 

observational data can also be considerable ^^^^^ it should 
a number of. written comments. In defense prsysce j-i is time we ii spent, 
be noted that the time spent in observing unit activates is ttme^ £ ciitiiC y 
The loss of efficiency is more than compensated for by the gam in tne 
of the information obtained. 

Another disadvantage of usteg s^ste^tie obse '|»|*y^ h ^f ^If!" 

observer who faithfully records what he/she sees ^ te?«a|ly obse ^ - 

familiarized with each item on^the ^--f^^f ^dbacfdaL. iLy will also 
completing the worksheets prior "collecting y their interpre- 

cess of systematic^bservation as « method of gathering feedback. 



Even with training, not everyone makes . good observer ^pme °f ervers^ 
lack the ability to adequately express their ^f^f^ecOrd^heir observe- . 
they can explain verbally w£at fg.^f ^observers, who possess the 
tions as written comments on the " orks ^- " ions of what they saw or their 
necessary writing skills, write their taterpretations of V ^ still 

conclusions rather than.sinply ..jecQrding w^t they actual y^ On- \ 

dtleiplined observers are therefore « necessity. 

SystSffiS tio observation r„,,:ire„ earef„. p.«nnlns end coordinatlon^ith the 
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bh certain events within a specified time frame. It involves coordinating with 
the observed unit to insut i that the observer has free access to the sites and 
events of interest; It requires knowing when and where unit activities are 
occurring, so that an observe'* will be present tb cover important events; 

Observers are not always greeted by those being observed with open arms, 
particularly when the observers are perceived as evaluate rs who are there to 
Judge the unit or the trainers. Sometimes those observed complain that the 
observers are distracting and interfere with the activities being observed, 
thus preventing them from doing their jobs as they see fit. The complaint has 
some merit; the presence of an outside observer cannot help but be somewhat 
distracting and may affect the conduct of the activities being observed. The 
extent to which events are conducted differently than they would be in the ob- 
server's absence adversely affects the accuracy of the feedback gathered. But 
the effect on the event observed may be positive if the event is improved because 
it was observed. Occasionally, attempts will be made to bar observers from cer- 
tain kinds of events. The reason given for excluding the observers may vary, but 
the result is the same — less accurate feedback. 

for Use 



Most major feedback collection efforts, should employ systematic observation 
as one of the methods for collecting feedback. The tendency to rely solely on 
subjective data may be reduced through the judicial application of observational 
techniques. As a check on the data provided by other methods and for a number 
of other purposes, systematic observation is an under-utilized tool that Has 
great potential for increasing the quantity and quality of information available 
for feedback. To realize the full potential of systematic observation^ tore- 
thought, planning, and careful execution of the method are required. Specific 
guidelines for using systematic observation for collecting feedback are provided 
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1. A form or worksheet should be designed to be used in making the obser- 
vations. The worksheet should contain a listing of the items or events to be 
observed. The items should list events as they might be expected to occur under 
ordinary circumstances. The kinds of items included on the worksheet will of 
course depend upon the particular activities being observed. Item listings for 
various activities are included in the references under "Observation ^Appen- 
dix D. Examples of items useful in evaluating training an ^ ^^tenance actiyi-- 
T&Tre ihbwn in Appendix C. Oth^rHL^may be developed for specific ^pljca 
tions bur: all items must meet certain minimum requirements The items should 
deal with thing, or activities that can be directly ob served Thin gs should be 
described in terms of observable properties, and activities should be described 
In terms of observable behaviors. To facilitate the recording of observations, 
the items should be stated so that they can be answered Yes or No with addi- 
tional space provided for more detailed comments. Examples of wor ^sheets that 

Uow these guidelines are included in Appendix 6 The ..*g^Jg«f ^SlS 
sheets included in this appendix are not sacred; they can be modified as needed 
to ^ke it- easier to record observations. However, we must adv ise ag -^mak- 
ing any changes that make it necessary for observers to record their subjective 
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judgments or evaluations of events. Avoid items that require the observer to 
rate each activity or to judge aspects of performance or the environment as 
satisfactory or unsatisfactory. Observers should use comment space for pro- 
viding additional details about what they observed. This space must not be 
used for recording subjective judgments or opinions about personalities, the 
unit, of unin activities. That is, observers should record specific behavior 
and facts that were observed and hot their interpretation of events. For 
example, they shouldn't record a general judgment that "the exercise controller 
didn't know what he was doing," but should instead record specific problems 
that were noted, such as scoring discrepancies or violations of SOP's. The 
iterat, on each worksheet should be preceded by a heading. The heading should 
identify the activity being observed, when and where the activity occurred, 
and who was present to inc lude the observer. /. 

2. All forms or worksheets developed for observing training, other ac- 
tivities or the environment should be pilot tested prior to using them to 
collect information for feedback purposes. The conditions under which the 
observation forms are tested should resemble as closely as possible those under 
which the forms will eventually be used. Typical observers should be among those 
testing the forms . The events observed, whether they be formal training events 
or day-to-day unit activities should also be representative of the kinds of 
events on which observational feedback will be collected. New observation forms 
may be piloted in conjunction with the training that/ each observer must undergo 
to certify that observers are knowledgeable about observational methods and 
the observation forms. By piloting the forms, it can be determined which items 
produce useful feedback and which observers are most proficient in gathering 
the feedback. , 



3 if more than one or two observers will be involved in collecting the 
feedback it may be advisable to conduct a workshop to train the observers. 
The observers should be told the. purpose of the workshop, i.e., to train them .... 
to use the observation worksheets for collecting information to be used as 
feedback. The importance of learning how to accurately record their observa- 
tions should be stressed. The instructor should go over each item to be included 
on the worksheet with the group to insure that they will know what i ; t is they_ are 
to look for, how they will know when it happens or foes not happen, how they . 

sho uld record their observations. The instructor should get feedback frpm^ the 
observers to insure that they understand the items ifr-tfie~same way, so that there 
will be standardization across observers. If a filmed Or £ i^f^Jgfg 0 nal 
sequence is available, the instructor can Tun the tape or film, 
those parts of the film that relate to the items on the worksheet A necessary 
part of the workshop involves giving the observers practice in using the 
sheets. The same events should be observed by all observers so that the observe, 
tions can later be compared. The events observed should be actual even s that 
resemble those on which feedback will later be collected The ^formation 
latnered during this practice session should be reviewed in class, and the «ork- 
Keets cbmpISId by the observers should be collected by the instructor, so that 
the instructor cai/determine which trainees record their observations most 
accurately and completely. References listed under "Observation" t. Appendix D 
provide additional guidance in selecting and training observers. 
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4. Those in charge of collecting the feedback should assure the unit or 
the trainers that the purpose of the observations is hot to evaluate thein^ but 
to assess the training program, task proficiency^ or the usefulness of training 
materials. Training developers arid evaluators should be open arid honest with 
trainers or unit personnel about their purpose for being there. Observers 
should not attempt to hide recorded notes or completed worksheets from trainers 
or unit personnel. Observers' should be cautioned not to interfere with the 
conduct of training, testing /or other unit_ activities* and should not interact 
with trainers of unit personnel any more tliari is essential to collect the neces- 
sary observations. At the same time observers must be told the importance of 
remaining close enough to the activities being observed to insure that the data 
gathered accurately reflect what occurred. 

5. Observers must be carefully selected on the basis of their performance 
during the workshop'. Completed worksheets should be reviewed in order to deter- 
mine each observer's proficiency in recording the required observations. The 
best observers are those who record what they observe in detail on the observa- 
tion worksheets, with a minimum of interpretation. Those 'who jump to conclu- 
sions or make sweeping generalizations on the* basis of what .they saw do not 
make good observers . 

6. Observers must go where the action is. To provide accurate feedback, 
they must observe important training events or unit activities first hand. If 
the. activities of interest are occurring in the motor pool or in a unit train- 
ing area, then that's where the observer must be. Arrangement should be made 
with trainers or unit leaders to allow observers free access to various sites. y 
Observers should have the option of appearing at these sites unannounced. How- 
ever, observers should take care not to disrupt unit activities by unannounced 
appearance? at sites- where-they -are, not known... Unit , p.exs^nn^jwst be informed, 

ahead of time that observers will be in their area from time to time, and 

should know generally what the observers will be doing. It may not be realistic 
for observers to arrive completely unannounced, but it is usually not necessary 
to announce when they "wilT "be where" of" tK^exaclTac t ivit x 

For scheduled events, such as scheduled training, observers should arrive in the 
area at least 10 minutes before the event is scheduled to begin. When^t he same 
event is occurring Simultaneously at several stations or locations, additional 
observers will be needed to provide adequate coverage of the event. 

7: Try to -rotaimize' the boredom and discomfort experien ced :5y_ observer^ . 

in planning data collection activities keep the observers 1 interests in mind. 
Avoid over-committing ^Onrvers by limiting the number of data elements that . they 
are required to collet at any one time and the hours, spent in. observation activ- 
ities on any one day. In addition provisions should be made for the well-being 
and comfort of the observers. Such provisions might include transportation to 
and from the site, messing facilities near the site and the opportunity to use 
them, and a temporary shelter where observers can go from time to tune to 
escape the elements. This will result in better motivated observers, and hence, 
more accurate and complete feedback,. 
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Testing 

As feedback^ test results may be used to provide objective informat ion 
about the skills and knowledges acquired during a particular trafaing prograi^ 
or to assess the proficiency of soldiers in the field in performing certain 
tasks. Training developers use testing to determine if the training P r ograms, 
training support materials* and on-the-job experiences result in the soldiers 
being proficient on specif ic tasks. Two types of test results may be useful 
as feedback: (1) Test results from tests that are a jsart of a training pro- 
gram or unit testing pre gram (i.e., results from existing tests or existing 

records)^ arid (2) Test results from tests specially designed^for feedback pur- 
poses. Because evaluators and training developers are primarily interested 
(or should be interested) in performance on specif ic tasks under a given set 
of conditions and standards, most test results used for feedback consist of 
records of hands-on performances. To the extent that hands-on tests, are not 
included as a normal part of in^pitutionai and unit testing, feedback collectors 
will need to design special tests for obtaining feedback on task performance. 
Considerations involved in using results of existing tests or designing special 
tests to gather feedback -will be discussed further in ^he guidelines for use 
of testing. 

Positive ancL ftegatj^yg Aspects 

As a source of feedback, test results are probably more highly regarded by 
training developers and evaluators that! any other type of information. When T 
asked how much confidence they place in the accuracy and reliability of various 
types of feedback they collect, training developers and evaluators typically re- 
serve-their-highest^^ — This-h-i^h-deg-ree^f-con^-idence-ma-^— 
be attributed to the perception of testing as a rigorous objective method for 
assessing performance. While this perception probably is accurate when tests 
. are conduct ed^pjroperly ,_the confidence is misplaced for poorly administered tests 



Some comSm problems that may affect the accuracy and reliability of tests, 
include: testing the wrong tasks; vaguely stated test instructions; tasks tested 
using inappropriate or unrealistic standards, and under wrong-conditions; and 
erroneous- test results due to biased or inconsistent scoring among examiners. 
Because test administration is rarely observed, such problems often go undetected 
and qu^^ienabl-~Jieat__results are accepted as accurate indices of performance. 



When tests are" conduct-^ 
Sore objective methods for obtaining feedback. The objectivity of tests makes 
them acceptable measures of training effectiveness, and permits them to be used 
as criteria against which other sources of feedback (e.g., estimates of task 

proficiency, Questionnaire responses, and interview Results) are compared. 

Tests administered before and after participation in unit training activities or 
institutional training programs may be used as one index of the effectiveness 
of that training. For checking on the accuracy or various estimates of task 
proficiency, no better criterion exists for comparison purposes than hands-on 
tests of performance . 
i 
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Testing, particularly hands-on testing, has not been the preferred choice 
of training developers and ©valuators for collecting performance information. 
The reasons for this are many. Foremost among these is the heavy demand placed 
oil the resources of trainers, units, and feedback collectors alike by hands-on 
tests. All'* feedback methods require the expenditure of time and resources by^ 
feedback collectors, and probably by those from whom the feedback ^is collec 
Of the various methods, hands-on testing requires the highest expenditure 
resources by collectors and suppliers of feedback. A resource that test ill g 
consumes in large quantities is time — a resource that is always in shorft sup- 
ply. In terms of personnel resources, hands^n^^strs^tfcn^ sample of indi- 
viduals to be tested, one or more ^tbia^edT^tr^ ined examiners, \somebne to make 
arrangements for provision of the^jte^Sed facilities and equipment*, and possibly: 
observers to monitor the administfratioh of the tests. Other resources jrequ irec^S^ 
for testing are carefully designed tests and scor.esheets , adequate testing " 
facilities, and equipment to be used in the. testing process. The large resource 
requirements of hands-on tests probably explain why: they are not a larger part 
of the feedback picture than they are presently. The resources required in 
testing depend to a considerable degree on. the number^of individuals orTgroups , ; 
to b* tested. -j&eilSaefc-^ 

test results are to be useful for feed^cV purposes. .While so^-ipplications 
of feedback do require that most or all of the individuals participating ^in an . \ _ m 
activity be tested (e.g. , in evaluating training effectiveness), testing a s^all 
sample of individuals can sometimes provide suf f icient information,. To discover 
which tasks are performed well by a unit, a representative sample of unit per- 
sonnel can be tested on a set of tasks. For spot-checking the accuracy of data 
obtained through the use of other methods, testing a small sample from the group 
from whom feedback is being sought is often sufficient. - 



Guidelines for Use 



\J. If the feedback desired consists of information about proficiency in 

perfoTming--specif^c-te^ 

, _ mu A *.>*«i*« nai a« o.a -F/-\-r- fVio focf choiil H he those that 



formance of these tasks. The tasks selected for the test should be_ those that 
the soldiers tested might reasonably be expected to perform on the basis of their 
training and experience. The tasks tested should be stated in terms of measurable 
performances, with conditions and standards that are consistent with those speci- 
fied in the training objectives or in the Soldiers Manual. Test instructions 
should clearly state what the soldiers are expected to do and s hould be presented 
-in^ar-st-andardi^ed maiiimi such LhaL each soldier receives the same instructions^- 



— ill — e£ — yi^IlTutlXu X6 cu mcuinc;*. nuu" w « " — : - 

<ir-n-re> ch^t-^ nVihiil ri be designed so that the examiner can record performanc 

efficiently and objectively. Distinct parts of the task that can be^readily 
observed should be listed on the score sheet, so that examiners can record, in 
addition to an overall evaluation, those parts of the task that were or were not 
performed correctly. Scoring each distinct observable part of a task provides 
additional information about that task which may be used by training developers 
or evaluators to pinpoint the reasons for poor task performance. There are of 
course practical limitations on the amount of detail that an examiner can record 
for a ny given task. But efforts should be made to record as much detail as- pos- 
sible when the data are to be used as feedback. 
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2. Testing requires the full cooperation of trainers in the un*^??.?* tlie 
institution. Arrangements must be made well in advance to s ? cil *e the needecj 
equipment, facilities, and personnel for the period during which the testing 
is to be conducted. Testing is an activity that requires careful planning and 
execution by those conducting the tests. Because the planning and execution - 
typically involve the trainers in some capacity* it is essential to enlist _ 
their cooperation. . Cooperation of unit and institutional trainers is more like- 
ly if the test results are not perceived as a threat. ^ If institutional, and unit 
trainers think tiiit test results collected will be used against them, they may 
refuse to permit training developers or evaluators to conduct the tests. _ Oips y 
way to prevent testVxesuits from being used inappropriately is to inaihtaih the 
anonymity of individuals and units in reporting results. Anonymity becomes more 



important as the level at which the results are reported increases.. Because 
training management is, accomplished at the battalion level_, generally there is 
no reason to report' results to levels above this level. _ When reporting results 
to the battalion commander, it is recommended that the companies not be identi- 
fied in reporting the results. Similarly platoons should not be identified in 
rej>6rt-ta^res^ 



be advisable to report individual. test results by name. While obtaining the 
cooperation of the unit may entail sharing the test results with unit leajders, 
it is important not to_ alienate. individuals by identifying those who performed 
poorly on the tests.; When results are to be reported outside of the battel ion, 
or used by feedback collectors* all references to individuals and units should 
be omitted . 

3. Proper test Administration is essential to obtaining accurate test re- 
sults. The~te_sts should be administered by independent, unbiased observer? 
whenever possible. This pres ents no projOemjfhen the tgsts_are__deyeiopgfcyfl- — 

-a? ^91Steria"^ByTrain in g develop er s and evaluators whose primary purpose is to 
obtain feedback . However , when tests are administered by those who conduct the 
training at the institution or in the unit, there may be a tendency towards 
leniency -in conducting and_s_coring the Jies ts . When ^ ^^^re_r^ admin ist^red_ 
by independent examiners, trained observers _from the evaluation team should 
observe the testing to insure that the tests are administered properly. A form 
for making test observations is included in Appendix C. Additional guidance in 
observation of testing may be found in references included under "Observation 
and "Testing" in Appendix^D^ — "~ 

4. Performance tests should be used to check on the accuracy of results 
n hf> a mpH thrbiig h-^^g-c^>^ Soldiers and their supervisors are 

often asked to provide estimates of how well they caSTperfOiw specific tasl«H 

Several precautions should.be taken in obtaining such proficiency estimates. 
The performance being appraised should lie stated as . precisely and completely 

as possible: Bating scales should be phrased in terms of explicit observable 
measures of performance rather than in general terms. such as 'average 1 , below 
average"* etc. Raters should be trained and have had experience with the _ 
rating scales and tasks being rated. As a further nfeasure, the accuracy of 
these estimates should be checked by selecting a sample of the^tasks and test- 
ing hands-on, the performance of individual soldiers on the tasks. At the 
time the proficiency estimates are obtained, the soldiers and their supervisors 
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should know they will be tested on selected tasks, but they> should not be told 
exactly which tasks are to be selected, tSless proficiency estimates are 
shown to be relatively good predictors of hands-on test perforaance, such es- 
timates should not be used for feedback purposes. 
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SECTION III - INTEGRATION OF METHODS 



Now that each available feedback collection method has ybeen discussed; 
this section addresses various issues related to the planning arid conduct of 
feedback activities. The gathering of feedback is a flexible process that 
must be adapted to the requirements of particular situations. None of the 
feedback methods discussed in this chapter ds sufficient in and of itself for 
gathering feedback in _al IV situations which may arise. A systematic approach 
is thus heeded in which methods, are integrated and tailored, to meet tihe heeds 
at hand. There is no set fornjulaor procedure, through which ah appropriate 
mix of methods can be selected and applied. There are general approaches, 
such as the sequential mix. mentioned earlier in this paper. In this approach - 9 
initial indications of problems are obtained through informal feedbackjor 
review of existing performance records. The^scope of problems identified is 
then determined through the administration of questionnaires, and further de- 
tails are obtained during follow-up interviews . If further in format ion appears 
to be needed at this point, it may be obtained through administration of per- 
formance tests or conduct of structured observations using preceded worksheets 
indicating what is to be observed. While such a general approach has the 
advantage of leading to the collection of more precise data as needed, it may 
not be applicable in all situations. Resource arid dime constraints may not 
allow the sequential application of all available methods. Informal feedback 
and existing record^ should not be solely relied upon for initial indications 
of problem areas, since problems may surface during interviews or observations. 
Feedback gathering should move toward the collection of more detailed and pre- 
cise data as resources allow, but there is no set sequence for the application 
of methods. The selection of feedback-methods should be based upon prescribed 
criteria, and some of these are delineated below. This is followed by discus- 
sion of other general issues, including sampling considerations and the struc- 
ture of feedback collection efforts. 

Criteria for Method S e lection 



There are numerous factors which interdependentiy impact upon the choice 
of feedback methods and the design of a feedback collection exercise. Among 
these are the resources available, the established accuracy of the methods 
available, and the scope of the area being addressed. Therfe factors should be 
jointly considered as criteria for making decisions oh hbii to collect feedback. 

The appropriate resources for application of each feedback method have 
beci addressed ih previous. sections, so, only general resource guidelines will 
be ofiered here. Most DTD's and DOES' s do hot. have sufficient personnel re- 
sources to support l&rge^schle, continuous collection of feedback. The acquisi- 
tion of feedback must thus be efficient in order to be doable within available 
resources. But- the feedback process. should hot be short-changed; a minimum 
amount of resources- is heeded. to collect adequate feedback. / Training develop- 
ers and evaluatbrs cannot collect adequate feedback while sitting in their 
offices receiving sporadic, informal feedback and mailing questionnaires. As 
discussed earlier^ informal feedback is usually very general and mailed ques- 
tionnaires are frequently hot returned. Training developers and evaluators 
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% must have the resources available to collect feedback first -hand by visiting 

training sites in the institution and in the field arid by interacting directly 
with users of products of the training development process. Training developers 
should thus riot try to collect feedback using just their own limited resources^ 
but they should work through and interact with Branch Training Teams (BTT's) 
arid other training evaluation teams. The combination of DTD arid DOES resources 
may help eliminate problems in resource shortages^ so it should be actively 
encouraged. If such pooling does riot provide adequate resources, for collecting 
needed feedback, then the scope of the area being addressed or the questions 
being asked should be reduced. For example, feedback may be needed on the: 
performance of all tasks within a given MOS, but resources may not be available 
to support the collection of such extensive data at brie time. Managers of feed- 
back effort* should acknowledge the existence of such a situation and tailor 
their efforts to collect specific data bna sample of the tasks of interest 
rather than collecting general data on all tasks. Precise data bri a subset of. 
tasks is likely to be more valuable and useful than general arid perhaps inaccur- 
ate data bri a wide range of tasks. This issue is further addressed below in a 
discussion of the accuracy arid scope of feedback approaches . 

Resource concerns relate not only to the number of personnel involved in 
feedback collection but also to 'their qualif icat ions and background. The ideal * 
feedback collector should have experience in both the subject matter being ad- 
dressed and in the collection methods being, used. Subject matter experts do 
riot automatically make good feedback collectors . ^ ow l e ^g? of the subject 
raattar being addressed is helpful, particularly in the gathering of specific 
details. This is brie of the reasons that training^ developers should be ±n T 
eluded bri feedback collection teams. But expert^ knowledge can sometimes be a 
hindrance. For example, during structured observations subject matter experts 
may become so intensely involved in the information being presented that they 
miss seeing major problems in how it is presented. Feedback collectors should 
be given training and practice in the application of the methods being used, 
especially interviews and observations^ They mus t learn to record the data as 
presented, arid not their interpretations of the data. A workshop in whi<-h the 
actual data collection forms are discussed and used is the best way to train 
data collectors. Guidance fox conduct of such a workshop is included in the 
references at Appendix B. 

As pointed out above, it is very important that the feedback process be 
efficient. But one consideration that is even more important is that the data 
gathered as feedback be accurate. Since major revisions in training programs 
and training support materials are based upon feedback, this information must 
accurately represent the needs and experiences of usars. Concerns for accuracy 
should dictate the ways in which available resources are utilized. Subjective 
data having unknown accuracy should always be checked against relatively ob- 
jective data. If a questionnaire or interview is used which has not previously 
been validated, then it should be supplemented by structured observations or 
tests. Validation of a questionnaire or other data collection form involves^ 
more than just the administration of . the form to a sample of personnel and the 
noting and correction of general problems in wording or format. A form is 
validated when the responses gathered using it have been shown to agree with 
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data gathered using more. objective, criteria. Since few if any feedback collec- 
tion forms have been validated* all feedback efforts should employ objective 
measures such as structured observations and testing at least zo a limited ex- 
tent. It may not be necessary, to chepk the accuracy of every/ subjective re- 
sponse obtained^ but spot checks should always be conducted^/ If such checks 
indicate problems with the accuracy of questionnaire of interview responses \ 
theh these responses should be disregarded arid replaced with more objective 1 
data. All available methods must thus be integrated in/4 feedback collection 
effort, with the degree to which each method is included in the mix depending 
upon the demonstrated accuracy of th^ data obtained. Sufficient resources 
must be committed to insure the collection of accurate feedback. If this is 
not done, inappropriate revisions , may be made in training, and the feedback' 
can do more harm than good. When resources are limited, the scope of the j 
effort should be reduced rather 'than, the precision of the data. Absolute. \ 
accuracy of feedback can never ^be achieved, but the highest possible level of 
accuracy should always be sought. j 

.... . / : - -/ - \ 

The discussion above has indicated ways in which considerations of scope 

interact with other factors impacting upon feedback method selection. _ Resource 
and accuracy concerns may limit the scope of a feedback collection effort. In- 
versely, the scope of an effort influences resource and accuracy cons iderat ions . 
If feedback is needed on one small segment of a training program, it can proba- 
bly be collected by one person within the Center/School without detailed accur- 
acy checks. But if feedback is needed on major portions of a training program, 
it must be collected by a team of trailing developers and evaluators visiting 

.f ieid_and_gatheririg_yarious types of data as accuracy checks. The scope of; 
the effort must be delimited during planning of feedback collection, and Re- 
sources must be made available which realistically allow data collection bf that 
^ scope to be accomplished. Otherwise, the quality of the data collected will 
likely prove to be unacceptable. 
i 

Sampling 

Another aspect of feedback collection related to the factors discussed 
above is sampling considerations. It is generally not appropriate to base feed- 
back on just one source, as is**somet imes done in informal feedback collection. 
But it is often not possible to gather- feedback from all users of a training 
product or on all aspects of a training program, due to resource and time limit- 
ations. After selecting the appropriate methods to be used, training developers 
and evaluators must then select the appropriate samples to which they are to be 
applied. Sampling relates not only to the specification of individuals and 
units from whom information is to be collected, but also to the specif icat ion of 
tasks and areas of concern which are to be addressed. In many cases sampling _ - 
can be made more efficient by integrating these two dimensions. That isj differ- 
ent samples of individual^ can provide feedback on dif ferent sets of tasks or . 
areas of _ concern. Particular sources may provide only part of the information 
needed, but the overall data collection effort should provide all the information 
heeded. 
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Sampling is not usually a concern in the gathering of informal feedback^ 
since such feedback is generally unsolicited and is passively received by 
training developers and evaluatbrs. But sampling considerations are important 
with all other feedback methods, especially those that are relatively resource- 
intensive, such as interviews, observations, and testing. Sampling of feed- 
back is basically a balancing act between obtaining enough data* but not too 
much* Sufficient data must be obtained to accurately represent the overall 
situation, but resources should not be consumed by collection of large amounts 
of data which may.be redundant and difficult to manage. Statistical sampling 
procedures often cannot be applied in feedback collection , since real-world 
constraints often preclude truly random sampling. It is thus difficult to 
specify the size of the sample needed in each feedback collection effort. For 
this reason, only general sampling guidelines are offered below. But this 
does not mean that training developers and evaluators should hot be concerned 
with statist ical sampling procedures. Sampling should always be conducted as 
ca refully and rigorously as the situation allows. Statisticians arid sampling 
textbooks should be consulted to supplement the guidelines offered here. 

To support overall evaluations of training programs and support materials, 
feedback should be gathered from more than brie unit or a small set of units in 
one geographic area , Contingencies such as range and ammunition availability 
vary in different locations, and training programs are often implemented some- 
what differently in different units. To obtain an overall picture of the situ- 
ation in the field, feedback should thus be sampled from several units in each 
of several geographic locations. Feedback obtained within a unit should be 
gathered from all levels of personnel involved in the training process. For 
example i if a questionnaire' is administered within a battalion on the utility 
of particular training materials , Kit should be administered not just to the 
senior leadership, but to all levels of users. Feedback on products of the 
training deve lopment process shoulti\ be obtained from personnel directly using 
those products, including Neb's s^r^ing as trainers and soldiers receiving 
training. When individuals are asked\to provide feedback through a question- 
naire or interview, they should not be ^siced to provide so many responses that 
they become fatigued. When respondents become fatigued, the quality of their 
responses is likely to suffer. Questions should be sampled so that no indi- 
vidual is required to participate in a survey or interview for more than one 
hour. As suggested above, different samples of questions can be addressed by 
different individuals in order to provide the complete feedback needed. 

Sometimes it is practically impossible to specify the size of the sample 
needed before embarking on a feedback collect ion effort. In these cases a use- 
ful strategy is to perform preliminary analyses of the data while they are being 
collected and to stop collecting further data when they become redundant arid 
provide no new information. Collectors of feedback should not expect to obtain 
every piece of relevant data that may be available.. But they should b! aware 
of general sampling considerations and should sample sufficient data to be 
representative of the situation existing in institut ional or unit training. 
Decisions on sample sizes needed are best based on experience with feedback 
methods and with the types of data collected. 
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Structure of Feedback Collection 

In this section the information In previous sections is S ummarized_jji 
terms of the general structure needed in feedback collection efforts. The 
first point that should be made is that feedback collection should be planned 
in advance arid structured as highly as possible. Specif ic uses should be 
planned for all data that are to be, collected, and sufficient copies of struc- # 
tured forms should be available for recording of each type of data. The re- 
sources needed should be organized, and the samples from. which data are to be 
collected should be clearly identified. Ail feedback collection forms should 
be thoroughly pilot tested, and all data collectors should be trained in the 
use of these forms. 

Another key point is, that it is very difficult to over-coordinate the 
collection of feedback. Thorough prior coordination will eliminate many prob- 
lems that commonly occur during data collection. Training developers and eval- 
uators should continually coordinate their efforts to avoid duplications and to 
make certain that the needed data are obtained. Coordination with those units 

or individuals providing the feedback will help insure that structured data 

collection schedules can be adhered to. Feedback collector s ^teul^etrtei: un ±ts~ 
looking for specific events or results, rather than looking for whatever hap- 
pens to be going on. This requires careful coordination with unit personnel, 
since unit training schedules are highly dynamic. Coordination with the pro- 
viders of feedback should not stop once the data are collected, but should 
continue as long as useful information . is being transmitted. Feedback providers 
should be informed as' to what conclusions were reached from the data they pro- 
vided and what resulting actions took place. They can then provide feedback on 
the effectiveness of these actions, and a continuous cycle u of feedback can be 
established. 
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SECTION IV - MANAGING FEEDBACK 



Feedback collected by any means is valuable only when it can be used to 
make informed decisions which lead to needed changes in training programs or 
materials, in addition to the level of detail and accuracy of feedback, the 
form in which the data are available and the _ speed and ease with which they 
can be accessed are important determinants of how feedback will be used. Rapid 
access to data that are in a readily usable form is dependent upon the instal- 
lation of an integrated data management system. Such a system is essential for 
organizations such as DTD' s ah d_ DOES 9 s who collect and use feedback on a regu- 
lar basis. Differences among these organizations in the kinds of feedback 
available and the ways in which they are used make it impossible to specify in 
detail the characteristics needed in a data management system. But systems for 
managing feedback should always incorporate certain general features in their 
design and follow certain guiding principles. These features and principles 
are discussed in ' the remainder of this section. 

. _ ______ _ / 

Automation of Da 



A primary princ iple for guiding data management is that it should be auto- 
mated to the maximum extent possible. Although computer support is not necessar- 
ily required fpr^ef feet ive data management, the increased efficiency and flexi- 
bility prbvided^by computers strongly recommends their use. The increased 

efficiency results from computers 1 ability to analyze large volumes of data 

quickly and to provide instantaneous access to data of interest. Because com- 
puters cin be programmed to quickly reorganize information in different forms 
(e.g., graphs, tables, etc.) and to extract and compare any data of interest, 
computers provide training developers and evaluators with the flexibility heeded 
to get the maximum benefit from the data they collect. 

The advantages of automated data jnanagement far outweigh any arguments that 
might be raised against the use of computers for that purpose. In the past,- the 
use of computers for widespread applications in the management of feedback may 
have been rightfully considered as too costly. Now it is not only, cost-effective 
to use computers for such applications, but it may actually be more costly not to 
use them. An elaborate computer system will not be; required to manage feedback 
at the typical Center/School; minicomputers are available which can accomplish 
this task with the necessary supporting equipment. A minicomputer with expanded 
data storage space, peripheral input/output terminals, and telecommunications 
support 'or interaction with other automated data systems should meet the feed- 
back management needs of any Center/School. Such systems should be standardized 
to some extent across installations to ease the transfer of data among them, but 
they should be flexible enough to meet the needs of each DTD and DOES; 

Because computers allow .easy access to data, they sometimes present security 
problems. Such problems are not* unique to computers, but occur whenever large 
volumes of data are maintained, some of which are classified or of a sensitive 
nature. Special techniques are available for securing information stored in 
computers, but j the need for implementation of such techniques can be eliminated 
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by avoiding the use. of sensitive or classified information as feedback^ infor- 
mation on task proficiencies^ training effectiveness, and usefulness of training . 
materials is generally not Classified, and it is usually not considered sensi- 
tive when unit and individual identifiers have been removed. These types of 
information comprise the bulk Of data that are used for feedback purposes. The 
benefits gained from the use of sensitive information for feedback purposes are 
generally hot worth the additional workload required to maintain it. . For this 
reason s it is recommended that the entry of classified or sensitive information 
into a feedback data base be minimized, if not completely avoided. As suggested 
Earlier! the sources of feedback should be kept anonymous in order to reduce 
sensitivity problems with the data. 

Preferably a computer dedicated to the maintenance and analysis Of feedback 
should be located on-site at each DOES. DOES personnel would be responsible for 
determining what information is included in the data base, but the data stored 
in the computer should be readily accessible by DTD personnel and other training 
developers. Remote terminals linked with the computer should be located at DTD 
and other appropriate locations, permit ting- direct access t o feedback hy rrain^ — 
ing developers. The data stored in the computer should be limited to feedback 
that is known to be accurate and has a clearly defined use. Care should be 

taken to insure that the computer does not become a depository for whatever 

large pools of unvalidated subjective data that are available. DOES personnel 
should maintain knowledge of the methods used to collect various types of avail- 
able data and should filter out data which have not been shown to be accurate 
or which do not have discernible utility. 

Many of the characteristics needed in a feedback management system may be 
found in existing data management and analysis packages. These packages ™ay be 
used, as is, or adapted to fit the particular requirements of each Center/School. 
If existing packages are not totally adequate, then prototype data analysis and 
management techniques should be developed ba^ed upon existing data and expanded 
as more data become available. In selecting or developing a data management sys- 
tem, close attention should be given to how the data are organized for retrieval 
from the data base. The system should also provide some means for analyzing the 
data and arriving at decisions on the basis of the^analysis . A capab|lity^should 
also be available for presenting the data in alternative easy-to-use formats. 
The data management system should be dynamic so that the data base can be updated 
as new data become available. It should also provide a mechanism whereby changes 
to training programs and materials based on feedback are recorded and ^racked to 
determine their effects. Finally, a data management system "•J.'P'J* g£ c £- fi - 
should make provision for integrating its data base with relevant ^ °» J« 

systems and Centers/Schools. These needs are discussed in further detail below. 

Organization o£ Bat a 

A primary advantage of computer-based data management systems over manual 
systems is the capability to retrieve needed data quickly. ^ 
which the automated system performs this function depends upon the^manner in 
which the data are organized for retrieval. Feedback relating to training 
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materials cafe be organized according to the particular materials to which the 
feedback applies.. Feedback on ^prbf icieilcy can be organized by task and/or 
mission^ and feedback, on _ the effectiveness of training programs can be organ- 
ized by class br block of instruction. By identifying the publication,: tasks, 
or class for which information is needed, the feedback user should be able to 
find : the relevant data_. In order to pinpoint feedback more pre c isely , the ^ 
system should pirihit~ the" user to specify other parameters. For example, in 
order to allow retrieval of infolnnation gathered during a given time frame, 

the stored data must be identified according to when they were collected^ 

Similarly, if data for different types of units (e.g^, active versus reserve 
units) are to be analyzed separately , then dat^aNn>u£t be tagged as to. the type 
of unit from which they were collected. There mf*y ?i?°_ be a neec * to index data 
so that only soldiers having certain experience or training levels contribute 
data to particular analyses. It may thus be necessary to include demographic 
information dri the sources cf/ feedback in the data base. 

The number and types o/f parameters on which the data are indexed depends 
upon the needs of the trailing developers and evaluators involved. When data 
are retrieved fr om the data basest he output~~lh~otild include all~^e^evan^[fdefft^ ^ 

f iers that apply. _„.In__ addi t ion to the identifiers described above (e\ g*j t ime 

frame, type of unit) , the output should include information about the size and 
composition of the sample from which the data were obtained. The instrument or 
_^et-hod^sed-4i^^ w hi c h t h e da t a w ere retri*™^— 

should also be listed or briefly described in the output . The inclusion of iden- 
tifiers such as these is necessary when data are obtained from diverse sources 
using a variety of data collection methods and instruments. Identifiers should 
not be included to tfie extent that they take up more storage, space than the data 
themselves, but sufficient information should be stored to allow the organiza- 
tion and retrieval of feedback along several dimensions. In this way, users of 
feedback, will have the flexibility to select the precise information that they 
need. 

Integration .of. Data 

As suggested earlier, the methods used to collect feedback should be inte- 
grated within a total system- Similarly* the data collected should be inte- 
grated. Often one method is used for checking the accuracy of data, produced by 
another method, dfenerally, when two methods aimed at obtaining similar informa- 
tion produce largely, discrepant results, the data produced by the more objective 
method are given moreXcredence in deciding which changes should be made. When 
different methods yield^essehtially the same results, then more confidence can 
be placed in the data and^result ing conclusions than if they were produced by a 
single methodology. Usually data collected by different methods are in some- 
what different forms and cannot directly be combined for analysis purposes, even 
when they are mutually supportive , But when there is more than one source of 
feedback about a particular training program or document, it is advisable to 
examine and compare* data from all sources. Often one source will complement the 
information provided by another source, even if the data cannot be combined di- 
rectly. For example, results of a test given at the end of a block of instruction 
may indicate that the training was not effective, but give little indication as to 
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why. Observations made during the instruction may pinpoint the reason for pfaor 
performance as being too little practice oh steveral of the tasks. To identify 
the problem and the correctable' reason for it Aboth sources of information are " 
needed* in similar ways^ data produced by questionnaires* interviews* observa- 
tions; and tests may ail complement each other. 

Data coilected^ErSi different groups by the s^me method may also comple- 
ment each other. For example, information collectedvf rom persbnnel_at lower 
echelons may reinforce and expand upon that collect eo\ f r om personnel at higher^ 
levels. When information is gathered from more than oke unit of the same type v 
within a relatively short period of time, the data may W combined during 
analysis and conclusions may be extended to the units as\a group . When units 
are selected in a representative fashion, conclusions may\be generalized to 
other units of that type. Training developers and evaluatqrs may at times 
collect the same information in the same general way without: being aware of the 
duplication. Such duplications should be identified and the\ data should be com^ 
bined so that training developers and eyaluat ors can benefit I from all available 

informat ion . FbF~cert aih kinds <* f- ^£^dba^k~^ itegra t itig tne^gata- W1 1 h r.hrasti 

gathered by other Centers/ Schools may lead to "the 'iden tif icat ion o f problems 
existing across branches or even across the entire Army. Integrating data in 
JtJiis— wa^— wou-1^— requ-i-re-e^ ions among co m p uters located at djrf- 



ferent sites and a greater degree of standardization of data collection and man- 
agement procedures than presently exists. 

Regardless of whether the data are gathered by different methods, from dif- 
ferent groups, or by different organizations, the important point to remember is 
that related data should be integrated to the maximum extent possible. Informa- 
tion in the data base should be catalogued and cross-referehced with related 
available data. Thus, if both observational and test data are available in the 
data base for a particular block of instruction, any attempt to access one type 
of data will automatically remind the user of other related data that are avail- 
able. The manner in and extent to which data are combined, during, storage arid 
analysis will vary with the typesof information involved* but related data 
should always be compared and! combined to the maximum extent possible.* Such 
integration will generally increase the amount of information available to train- 
ing developers and e valuators, and this will lead to more effective decision- 
making based upon feedback. 

Data Analysis and Decision-Making ; 

The methods used for analyzing data depend on the Mnds of data collected 
/and the form in Which they are obtained. Responses to questionnaires or inter- 
j . views must usually be coded' so that each response is assigned to one of a 
snail number of categories before being entered into a computer for analysis. 
For example, if soldiers are asked what, activities other than training they 
participated in during the last two weeks, the responses given may vary widely. 
In order to .facilitate the analysis and reporting of the data^' the various 
responses could be assigned to categories such as the following: (1) mainte- 
nance, (2) p6it detail, (35 inspections, and (4) other activities. Sometimes 
the response categories are knoun in advance and can be preprinted on the data 
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collection form as response alternatives. In other cases it may be necessary 
to derive the categories after . the data have been .collected. The basic data 

iri.Jbbth .ca§es_ will consist of frequency counts of the number of responses given 

in each of the categories. 

As far as statistical analysis of feedback is concerned, the best general 
guidance that can be given is to keep the analysis as simple and clearly under- 
standable as possible. The purpose of analyzing feedback is to reach arid sup- 
port conclusions and recommendations about changes needed in training, programs 
and materials. Many of the personnel involved in making decisions about these 
recpmmendat ioVis do hot have_ an extensive background in .data analysis; it will 
thus be necessary, to keep the presentations of analyses simple arid straight- 
forward so that these individuals can see how the data support the conclusions 
reached. Also* the application of sophisticated analysis techniques frequently 
involves numerous assumptions about the characteristics of the data being analy- 
zed. Data gathered in an operational military environment of ten, do not meet 
many of these assumptions. It is thus best to cautiously apply sophisticated 

anal ysis techniques iti the processing of feedback, and to consult a Stat is ticjtan 

"or reference_text _,to make sure jhat. th 

The analysis of feedback may be limited to the descriptive level , in terms of 

frequency counts, percentages, or proportions. For example, the data of in- 

terest may be the percentage of soldiers failing to perform a given task to 
standard or the proportion of those responding "yes" to a particular question. 
Summary statistics such as means or medians can be used to concisely describe 
data collected, but in some cases use of such statistics may not be appropriate. 
For example, two sets of data ranging along a five-point scale may have the same • 
mean but considerably different distributions, it is thus prudent to plot the 
data graphically or- otherwise examine the distributions of data obtained, as 
well as calculate summary statistics, in analysis of feedback, one should stick 
as closely to the original form of the data as possible arid take care to insure 
that any summary statistics used are appropriate. 

Techniques of inferential statistics should be used iti making decisions 
about the significance of differences among sets of _ data ^ and statistical signif- 
icance should be used as a criterion for deciding whether or hot changes are 
needed in training programs of materials. However, statistically significant . 
differences are not always practically significant; thatis* the magnitude of the 
differences may not justify the time and expense required to change training 
programs or materials. For example, adding 10 hours of instruction t ; o a train- 
ing program may increase scores on the end-bf-course test by a statistically 
significant amount, such as from a 70 percent to a 74 percent GO-rate. However, 
the trainers may not feel that this 'y amount of improvement justifies the addi- 
tion of 10 additional hours of instruction. They may feel that an improvement 
of at least 20 percentage points is Reeded to justify the additional instruc- 
tional time. Techniques are available for selecting samples so that statis- 
tically significant differences are likely to have .practical significance; a 
statistician should be consulted during the planning stage of data collection. 
Even in the absence of such techniques, decisions about the implications of 
feedback should be based upon practical ,as--well as statistical considerations. 
Changes in training programs and materials should be made orily when the results 
of analysis meet criteria for both statistical and practical significance. 
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In sofce cases, practical decision-making criteria may ber somewhat arbi- 
trary, at ljast during initial iterations of feedback collection. Criteria 
may n e^^ o~b^setfeisr^ le 
range will indicate a problem serious enough to warrant modifications in _ 
training programs or materials. An exanple of such a criterion is the guide- 
line which states that if more than 20 percent of the soldiers tested on a 
task receive first-tii&e NO GO's, then the training program for that task is 
inadequatte and must be revise^. A criterion such as this can be entered into, 
a computerized data management^system so that values falling outside the 
acceptable, range are automatically listed, thus providing an easy way of 
identifying potential problem areas>_ Many of the criteria used in decision- 
making based on feedback are admittedly arbitrary^ but to; some extent so are 
those used in statistical analysis^ The test available decisionmaking approach 
is to use criteria which are determined ae objectively as possible, consistently 
applied, and modified based upon experience and feedback. 

One reaso n- for -id^ifying feedback by the time frame during which it was 

co 1 1 ected- i s -to-all-ow^cton ge s o ccurrin g - over fiine n i~t£tr-T 3 etet:fred^ 

* — — = - - . ____ gm „_ __ ____ _ _ *. -P^-v 



fication of trends\in data is a criterion that is often used as the basts__fpX- 
making changes in training programs and materials. But interviews of users of 

feedback ^ndicare tfraL Llm ideii t iflcatlon -of-- trends is- frequently based- on 

general feelinas that recent data represent a significant departure ±rom data 



general feelings that recent data represent a significant departure from data 
obtained in the pWt. If training is going to be modified on * - * baTis of 
trends, than the minimum requirement is to collect feedback on the area of con- 
cern on three or lore separate occasions and plot the results graphically as 
evidence that a /rend exists. Examination of the data using decision-making 
criteria such as those discussed above should indicate whether a sufficient 
trend exists to require a reaction. Statistical procedures are available^for 
testing the nature and significance of trends. However, the complexity of^ these 
procedures and the assumptions implicit in them limit their use in the analysis 
of feedback. If such procedures are used for identifying trends, a knowledge- 
able statistician should be consulted. It should also be noted that trends 
other than those occurring over time may be of interest during analysis of 
feedback. For •example, a trend toward increasing gurmery scores^ with increasing 
hours of practice on a gunnery training simulator and similar relationships will 
be useful information for training developers. The points made above for temper 
al trends apply to all types of trends of interest in feedback analysis. 

Reporting of Data 

The processing of feedback should not atop with the analysis and interpre- 
tation of results. In order to insure that needed changes are made in training 
programs and materials, data summaries and resulting conclusions and recommen- 
dations must be presented to appropriate decision-makers. Reports of the 
results of feedback collection*^ often in the past consisted of little more 
than a listing of the general problems discovered. To make a convincing argu- 
ment for needed changes, these reports should present, data to indicate the 
eSent of the problems noted, and they should offer specific recommendations for 
problem solution. They should also provide information about how the data were 
collected, how many units and individuals were involved, and the time period 
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dyer which the data were gathered, ^is^infori^ion 'should be presented in a 
clear concise manner* so that the changes needed and their probable impacts 
are readily apparent. / 



An automat ed data bas^ eapable of presenting information in a variety of 
formats can enKance the clarit^^conciseness, and efficiency of the data presen- 
tation. The data management system phould have the capability to quickly gener- 
ate line graphs^ histograms, and other simple pictorial representations of the 
data. It should, also be able to generate tables, statistics, -and summaries of 
major problems observed within a given time frame for inclusion in monthly or. 
quarterly evaluation reports. These reports could be composed and stored on *' 
the computer, making them readily available to users through interfacing termi- 
nals. Users could also create their own reports for special purposes by 
selecting the appropriate data* analysis options, and presentation formats 
through the terminal. To support this capability, an appropriate command from 
a terminal should access a catalogue of jttte various types of data, available in 
the data base arid the options and procedures available for analyzing and dis- 
playing them . Fro m this list t h e user could-serect~t"he type ^^^i =::: ^4 := meitad^ 
^^^gJ^si^JEo^4^-aropi:oy ccl A ^ ntTse quBu L "i iil^fll^r^^ used to select the 
time frame arid soldier population of interest. With such a system, the user 
could quickly retrieve the data needed in a form Jbest suited for particular 
applications. — — _ 

Follow-Up 

The gathering and use of feedback is a continuous cyclic process that does 
n 9? ? nd _ w ^ en _ c ^ n Sf s __ have _^ egn ^ n training programs or materials. Fbllow- 

ing such 'changes in response to feedback, it is important to measure, their ef- 
fect^ This usually involves collecti on of the same sorts of feedback which led 
to original identification of the nee3~for change. For example, if performance 
on the test given at the \conciusion of a course in map-reading increases notice- ' 
ably following the introduction of modificat ions to the instruction, then the- 
indication is, that the changes increased training effectiveness. The degree to 
which the improvement in performance can be attributed to the course modifica- 
tions depends on keeping other factors constant, such as the skill arid abilities 
of the soldiers being trained and the test content arid administration procedures. 
Follow-up is necessary to insure that changes instituted on the basis of feed- 
back are producing the intended outcomes. If negligible or unfavorable outcomes 
result, the feedback system itself may require modifications, since it is leading 
to ineffective changes. 

As indicated in the first section of this handbook, DOES personnel frequent- 
ly collect feedback and pass information along to training developers concerning 
problems with training programs and materials. Training developers are expected 
to take corrective actions, but they may not do so because they are not con- 
vinced that a change is necessary or they do hot- have! enough information to know 
what specific changes are required. The result is that feedback is not used and 
has rib impact. Unless the collectors of feedback actively follow up to deter- 
mine what changes are made and what recommended changes are not , they may 
incorrectly assume that the feedback they provided was completely satisfactory 
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and resulted in appropriate actions. The ultimate sol utio n to this problem 
^^Jj-^^ijj-^e-a^ievcn^at^ aiFcdirtlhwu^ between, 
training developers and eyaluators* _ Since such a situation does riot presently 
exist, collectors of feedback^ should follow up information that they have pro- 
vided to other agencies, in order to determine exactly what changes were made 
and their effects, A feedback management system must include a mechanism for 
monitoring* itself to insure that it has the desired effects. 
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ARTEP TRAINER QUESTIONNAIRE 



INSTRUCTIONS: The purpose of this questionnaire is to obtain information on 
your use of the Army Training and Evaluation Program for Mechanized Infantry/ 
Tank Task Force (ARTEP 71-2). The information _ydu provide will help insure 
that ARTEP 71-2 truly serves the heeds of field units. The questions below 
address the ARTEP document itself as well as the process by which you use this 
document. Provide answers based upon your unit's experiences in training and 
evaluation exercises planned and conducted by you or higher headquarters. 
Answer the questions for the level of unit to which you are assigned. That is, 
if you are a platoon leader or sergeant ^ answer for the platoon; if you are a 
company commander, X0 k or 1SG, answer for, the company; and if you are a bat- 
talion commander or S3, answer for the battalion. 

First provide the background information requested by filling in the blanks 
below. Then answer each question by circling the letter corresponding to your 
response arid writing brief responses, where appropriate^ More than one response 
may be appropriate for some questions; these are indicated beiowi No attempt 
will be made to identify or evaluate you or your unit^ so please answer each 
question as completely and honestly as possible. 

Date: Duty Position: Rank: 



Specialty Code/MOS: Primary . Secondary 



Months in Duty Position: Months in Present UnJ';: _ _ 

~~ — — -/v 

Last Institutional Training Comple'r^ti 

(AOB, ANeoe, etc.): Year Completed: 19 

1. What typfj(s) of armored vehicles does your unit have? 

a. M48A5s d. M551s g. M106s 

b. M60Als e. His h. M901s 

c. M60A3s f . M113s_ i. M3s 

2. Where is your unit located? 

a. Continental United States (CONUS) 

b. Korea 

c. Germany 

3. What component does your unit belong to? 

a. Active Army 

b. National Guard j 

c. Reserve 

- _\ 1 

4. Have you used ARTEP 71-2 or participated in an exercise derived from it 
during the past 6 months (circle both a and b t if appropriate)? 

a. Yes, in my present uniti 

b. Yes, in another uniti 

c. No. 

NOTE: If you circled c in question 4, do not complete the rest of this 

questionnaire^ since many of the questions ask about your experiences 
with the ARTEP in the last 6 months. 
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5- iri w^at form is ARTEP 71-2 available to you? (Circle as many responses as 
are appropriate;) 

a. I have my own copy of the ARTEP. 

b. I have copies of the pagefe^of the ARTEP that are relevant to my unit. 

c. I have cards or other papers that contain information from the ARTEP 
relevant to my unit. 

d. The ARTEP is in a central location where I can use it, (Where?) 



I_^ave been briefed on the ARTEP but do hot have access to written 
ARTEP materials. 

f. Other. (Please explain.) ' 



6. How often do you use the ARTEP materials that are available to you? 

a. Daily d. Once a year 

b. Weekly e. Never 

c. Monthly 

7. Which version of ARTEP 71-2 is available to you? 
a. New version published in November 1981. 

:_-=" __jb^^^ raft.-versibn pub lis h ed in March 1981, 
c. Earlier version. 

8. Have you received training in how to use ARTEP 71-2? (Circle as many 
answers as are appropriate.) 

a. Yes, in institutional training (AOB, AOAC, ANCOC, etc.). 

b. Yes, in a Battalion Training Management System (BTMS) workshop. 

c. Yes, from the chain of command in my present unit. 

d. Yes, other. (Please explain.) ''• 

\ 

\ 

\ 

\ 
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e. No. 
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9.__ If you have received training in use of ARTEP 71-2* did it prbyide you with 
sufficient information so that you could use the ARTEP in training your unit? 

a. Yes. 

b. No. (Please explain.) 

10. Is the terminology in ARTEP 71-2 clear to you? 
a. Yes. 

b; No; (Please explains) _ . _ . 

11. Is the terminology in ARTEP 71-2 clear to your soldiers? 

a. Yes. J 

b. No. (Please explain.) _ _ _ _.. 



12. Are there any missions or tasks hot included in ARTEP 71-2 which your unit 
must perform to be combat ready? 



a. Yes. (Please list them.) 



b. No. 

13. Are there any missions or tasks included in ARTEP 71-2 that should be deleted? 
a. Yes. (Please list missions and why they should be deleted.) ;; 



b. No. 

14. How many training and evaluation exercises from ARTEP 71-2 has your unit 
conducted or participated in during the past 6 months? . . _ 



15. How many of these training and evaluation exercises were planned and 
evaluated by: (Circle as many answers as are appropriate and fill in blanks) 



a. Platoon? 

b. Company? 

c. Battalion? 



d. Brigade? 

e. Division? 
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16 • How many of your unites training and evaluation exercises -> during the, past 
6 months, actually used OPFOR? ^ 



If OPFOR was not used, why not? 



17. How many of your unit's training and evaluation exercises, during the past 
6 months, used MILES? - — — - — 



If MILES was not used, why not? 

f 

I; 



18. How many firing proficiency/live-fire exercises has your unit conducted 
during the past 6 months? . _ -_ ■ 

Which ones; in what modft (dry-fire, MILES, subcaliber, f ull-caiiber) ? 



19. Which of the General Missions from ARTEP 71-2 has your unit integrated into 
training and evaluation exercises during the past 6 months? Circle "as many as 
applicable. 

a. Plan and control combat operations 

b- Maintain operations security (OPSEC) 

c. Perform tactical intelligence functions 

d. Cbhdu.- i: NBC defense operations 

e. Defend against air attack 

f. Conduct sustaining operations 

20. For your unit's training and evaluation exercises, during the past 6 months, 
who selected the missions and tasks (in terms of duty position)? _ . ; — 



23.. Ori the average, how much time have you had to plan each of your unit's 
training and evaluation exercises during the past: ' onths? ; . . 

If this has not proved sufficient how much time do you need? 

22. Do you use any documents in addition to ARTEP 71-2 in order to train to 
standard? 

a. Yes. (Which ones?) 



b. No; 

23- Did the conditions listed in ARTEP 71-2 match those your unit actually 
experienced in training and evaluation exercises during the past 6 months? 

a. Yes. 

b. No. (Please explain.) 



24. Was your unit trained arid evaluated in accordance with the standards listed 
in ARTEP 71-2? 

a. Yes. 

b. No. (Please explain. ) — ' " — — - - • ~ i"r::i""r 



25. Were you required to repeat tasks and/or missions until your unit performed 
to ARTEP standards? 

a. Yes. 

b. No. (Please explain.) — — 
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26. What type of feedback did you receive bri your unit's mission peirf brmance? 

a. None 

b. General performance ratings 

c. Detailed information on tasks performed well and poorly 

d. Other. (Please explain.) 



27. What type of feedback did you provide to your soldiers on their performance? 

a. None 

b. General performance ratings 

c. Detailed information on tasks performed well and poorly 

d. Other. (Please explain.) — _ 



28. Who provided feedback during training and evaluation exercises (in terms of 
duty position) for the following? 



a. Platoon 



b. Company 

c. Battalion 



d. Brigade 



e. Division 



SI 
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29; How much time elapsed between mission perf ormahce arid provision. of feedback 
for exercises, that were planned arid evaluated at the following levels? (Place 
checkmarks (/) in the appropriate boxes.) 

Immediate. 2-24 Ilrs 2-6 Days 7+ bays 

a. Platoon 

b. Company 

c. Battalion 

d. Brigade 

_ i ■ ■ » 

e. Division j 

J L 1 u — 

For what purpose does your unit most often use ARTEP 71-2? 

a. As a training tool 

b. As a test 

c. Other. (Please explain.) - — — — -_ _ 



31. List any other comments that you have about ARTEP 71-2: 



30. 



i . v ; 
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PROFICIENCY ESTIMATION QUESTIONNAIRE - 19K TANK COMMANDER 



Tbdayybiir Performance of the. tasks listed below will be tested. :In order 
to get a GO on any taskj you will have to perform all the steps on it p^gperly. 
The equipment and manuals you heed to perform each task will be available. 
Before you aire tested - % we want to see if you can ^accurately predict how wfell 
you will dp bh each task. Write, in v your namearid the other information irer- 
guested belbw^ and then predict how you will do oh the test by circling a _\ 
letter beside each task indicating your response. The letters stand for the 
following responses: \ \ 

Y — Yes, I can do this task arid will get a GO on it. \ 
? — X don't know whether I carl do this task or riot. \ 
N — No, I can't do this task arid will get a NO GO bri it* 

Name * — . Grade Unit 



How man> months have you been ah Ml TC? 



Perform bef ore-operations checks and services on an Y ? N 

Ml tank. 

Perform gunner's and loader's prepare-to-f ire checks Y ? N 

and services on ari Mi tank. 

Perform tank commander's prepare-to-f ire checks and Y ? N 

services on an Ml tanki 

Boresight the main gun on an Ml tank within Y ? N 

10 minutes i 

Boresight a caliber .50 M2 HB tnachinegun within Y ? N 

20 minutes. 



— Now we wa nt you to do "otre Wo"r e~ t Ki n g on t h is page . " Rah k how well you can 
do the five tasks above by putting the numbers 1 through 5 in the blanks to 
the left of each task. Put one number in each blank arid use each number once. 
Put 1 beside th^ task that you can do the best, 2 beside the task that you can 
do hext^best^ _and _so_bn__tol_5_beside the task that y°u expect to do the worst 
on. THEN GO ON TO THE NEXT PAGE. 



*N0TE: Names will_ be used only to match predictions with test results and 
they will be removed price the data are tabulated. The information 
you provide will riot be used to evaluate you or your crew members. 
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TC'S ESTIMATIONS OF GUNNER'S PROFICIENCY 



N'ow that you've predicted how you're going to do on the test todays we 
w ^nt you to predict how the members of your crew will do. On this page you 
are to predict | how your gunner will do. The tasks he will be tested on are 
listed below. ;Write in your gunner's name arid the other information requested 
below, and then predict how he will do dri the test by circling a letter beside 
each task indicating your response. The letters stand for the following 
responses: 

Y — Yes, ; my gunner cari do this task arid will get a GO on it. 

; — ^ do^t know whether r.y gunner can do this task or not. 

N — No, ray gunner can't do this task arid will get a NO GO on it. 

Gunner's Name - + 



How many months - has he been your guririer? 



Circle one: 



Perform bef ore-operations checks and services on Y ? . N 

an Ml tank. 

j 

Perform gunner's and loader's prepare-to- fire checks Y ? N 

and services on an Ml tank. 

Perform tank commander 1 s prepare- -b-f ire checks Y ? N 

and services on an Ml tank. 

Boresight the main gun on an Ml tank within Y ? N 

— - 10 minutes. • • ' : - -- u 

Boresight a caliber . 50 M2 HB machinegun within Y ? N 

20 minutes . 

Now rank hbip well your gunner can do_ the five tasks above by putting the 
numbers 1 through 5 in the blanks to the lef t_ of eac h task. Pu t a 1 beside 
"the" "task he cari"~do -best~i"2"j5lside the_task he can do next best^ and so on 
like you did for I yourself. THEN GO ON TO THE NEXT PAGE. 



TC f S ESTIMATIONS OF DRIVER'S PROFICIENCY 

Now you are to predict how well your driver will do on today T s test* just 
like you did for your gunner on the previous page. The tasks he will be tested 
on are listed below. Write in your driver's name and the other information re- 
quested below, arid then circle your response for each task. 

Y — Yes, ray driver can do this task and will get a GO on it. 

? — I don't know whether my driver can do this task or not. 

N — No, my driver can't do this task and will get a NO GOl on it. 

Driver T s Name 



How many months has he been your driver? 



Circle one: 



. Perform bef ore-operat ions checks and services on an Y ? N 

Ml tank. 

Prepare driver's station for operation on an Ml tank Y ? N 

within 12 minutes. 

Start and stop the engine on an Ml tank within 10 Y ? N 

minut es . 

Secure driver's station on an Ml tank within six Y ? N 

minutes . 

Perform after-operati o ns checks and services on an _ J$ ? N 

Ml tank. 

Set headspace and timing on a caliber .50 machinegun Y ? N 

within 110 minutes. 

Now rank how well your driver can do the six tasks above by putting the 
numbers 1 through 6 in the blanks to the left of each task. THEN GO ON TO THE, 
NEXT" PAGEt — — f 
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TC'S ESTIMATIONS OF LOADER'S PROFICIENCY 



test TC Ss f JSK i ? k prediCt well your loader will do on' today's 

rest.. The tasks he will be tested ori H ef . a 4 ii-i- — ^ * „ _ h r."° a y s 



Y — Yes; my loader can do this task and will get a GO on it. 
v -- I d ° n t 1 kn ° W whet V er ^ loader can do this task or not. 

N °' my loader can c d ° this task and will get a NO GO on it 



Loader's Name 



How many months has he been, your- loader? 



C4rcle one: 
;Y ? N 

Y ? N 



■ Perform bef ore-operations checks and services 

on an Ml tank. ■ - 

Perl^m operator maintenance on the 105mm breechblock 

assembly on an Ml tank within 14 minutes. 

— Jlear an M2 AO coax machinegun to prevent accidental \ 

discharge on an Ml tank within 30 "seconds. 

— In staH/remove an M240 coax machinegun on an Ml tank 
within 10 minutes. 

Perform operator maintenance on an M240 coax 

machinegun witMn 4 1/2 minutes. 

Set headspace and timing on a caliber .50 

machinegun within 10 minutes. 




N. 
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APPENDIX B 
SAMPLE INTERVIEW FORMS 



6?* 

ERIC 



INTERVIEW: TANK CREWMAN 

Date 

Position: GNR LDR DVR Months in Position 

Unit — 

1. How long have you been the (G, L, D) dri this tank?; - - -- - 

2. How long have you been an Ml (G, L, D)? . 

3. In the last two weeks, how many hours have you spent in: 

Scheduled Training „ : - - Unscheduled Training 

(Training refers only to those activities whose ptimary purpose is 
teach MOS related skills and knowledges;) 

j — - 

A. What tasks were trained? • f 



5. Were tasks trained to standard? Yes No 

If yes, what- were the standards? 



6. Who did the training? TC Pit Sgt Pit Ldr Othe r 
—7. Did you actually get to perform each task? Yes__ 

8. Were you told how well you did on e^ch task? Yes__ 

9. If you had trouble, were you given additional training 

on that task? , Yes_ 

10. What other activities took up time? 
- Maintenance 

— Inspections 

Details 

Other (specify) 
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- No 

- No 

- No 



INTERVIEW: TANK CPJEWMAN Date 

Unit 



11. Have yov used the bat ta lion ^ learning center in the 

last two typical garrison weeks? Yes No 



If yes^ what lessons? If no, why not? 



During duty hours Af ter duty hours 

As a member of your crew On your own 



Was there anyone there to help you? Yes No 

12. Have you used the MOS library in the last two 

typical garrison weeks? If not, why hot? Yes No 

13. Aire you a graduate of: BAT ... - _ 



(Which HQS, date) 

NET CONUS -- _ NET USAREUR 

(Duty position in which trained, date) 

14. What was your first duty position following graduation from BAT? 

GNR LDR DVR — ' Other 



15; Are you currently enrolled in ACCP? Yes — Nb_ 

If yes, what course? 

What Tther Aec courses have you taken? 



?If on tank, ask to see -3 
i'^esent? Used Appearance? Up to Date 



(Indicate Vol 1, V 

(If hot on tank) Do you have a -10 on the tank? Yes No 

In the last tu?b wesks v for which taslc.3 was the- -10 used? 



m 



INTERVIEW: TANK CREWMAN Date 

Unit 



17 ♦ Do you have your own copy of the SM? Ye s— No 

Vhat tasks have you looked at in the last two weeks? 



18. What problems have you had in driving, loading, firing, your tlank? 



19. What ^problems chave you had in maintaining your tank? 



/ 

/ 

20. What tasks should you have been trained on in BAT/NET that you weren't? 

/ 

/ 
/ 



v. 



ave you received training on these in the unit? / Ye s Nb_ 



If yes, which ones? / 

i 



21. Is there anything you would like to add? 



Interviewer 
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INTERVIEW: ARMOR COMPANY CDR, XO, aud 1SG 

Dat e 

Position: CDR XO 1SG Uni t,-- 

1. How long have you been in your present duty position 

in this company? ' 

2. a. Are you a graduate of: AOB AOAC 

(date) (SC, date) 

ANCOC MASTER GMR 

(MOS, date) (MOS , date) 

NET CONUS . _ NET USAREUR , 

(duty position in which trained, date) 

b. What other formal Army schooling have you had? 



3. In the last two weeks, how many hours has your company spent in: 

Scheduled training — Unscheduled training - — - 

Individual training Collective training 

(Training refers only to activities whose primary purpose is to teach 
MOS related skills and knowledges.) 

4. What activities, other than training, has your company been involved in 
during the last two typical garrison weeks? 



5. During the last two garrison weeks , how many hours did 
you personally spend supervising or monitoring training? 

71 
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INTERVIEW: ARMOR COMPANY CDR, XO, and 1SG 



Bate_ 
Unit 



6. What individual tasks were drained during the last two weeks? 



7. Were tasks trained to standard? 

If yes, where did the standards come from? 



Yes 



No 



8. Who conducted most of this training? 



9. How were these tasks selected for training? 



(duty position) 



10. During the last quarter, what collective training has your company par- 
ticipated in? Where was this training conducted? 



II; a. What problems came up during thi^s. training? 



b. What was done to address these problems? 
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ijjgg|VIEW: ARMOR COMPANY CDR, XO, and iSG 

Date 



Unit 



Do 


personnel in your company have: Trainer f s Guide 


Yes 


No 




Crefc Drills 


Yes 


No 




Soldier f s Manuals 


Yes 


No 




Job Books 


Yes 


No 




s i your company use written training plans? 


Yes 


No 



If yes, in what form (lesson plans, T&EO f s, etc. )? 



(ask to see one) 

14. What ether materials do you use to manage, conduct , or evaluate training 
in your compc ~y? 



15. What training support materials do you heed that you don't have? 



16. a« What are the hardest tasks for crewmen in your company to perform? 



_ i 

b. What makes these tasks hard? 



17. Do you receive satisfactory support from organizational 

maintenance? Yes Nb_ 

If riOf explain. 
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INTERVIEW: ARMOR COMPANY CDR, XO* arid 1SG 



Date 



Unit 



18. a. What percentage of your soldiers have transferred 
into the company during the last quarter? 

b. What percentage of your soldiers have changed duty 
assignments within the company during the last 
quarter? 

19* What are tlis operational constraints on training in your company? 



20. a. What are your training goals for your company? 



h. What problems hav^i yew had in meeting your goals? 



c. What can the Center/School do to help you meet your goals? 



21. a. Do you participate in battalion-level planning and 



dec is ion-making? 



Yes 



If yes, in what way? 
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INTERVIEW: ARMOR COMPANY GDR, XO, and ±SG 

Date 
Unit 



b. How do you involve subordinates in company-level planning and decision- 
making? 



22. If you could change anything you wanted, what would you change to make 
training better? 



23. Is there anything else you would like to add? 



Interviewer 



APPENDIX C 
SAMPLE OBSERVATION WORKSHEETS 
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TRAINING OBSERVATION WORKSHEET 
Name or Type of Trainin g _ 
Trairier(s) 



Date 



Unit 



Trainee(s)- 



Training Site 



Training Started (Time) 



OBSERVATIONS 



10; 
11. 

12. 



Were the soldiers, told the training 
objectives to include tasks, condi- 
tions arid standards? 

Did the trainer define new terms and 
identify new equipment for the 
soldiers? 

Did the tra incr demonstrate how the 
tasks and subtasks are performed? 

Was the demons t ration conducted in 
such a way that soldiers could see^ 
hear and comprehend what the 
trainer was doing? 

Did all soldiers practice each task 
and su'Jfksk? 

During Practice, did ail soldiers 
perf orm the tasks to st andard 
unassisted? 

What steps gave soldiers the most 
trouble during practice's 

Did every soldier practice under 
the direct supervision of* the 
trainer (s) ? 

Were soldiers told what they were 
doing right or wrong during practice? 

What training equipment was used? 

Was there enough training equipment 
for the number of soldiers being 
trained? 

Did the training equipment work 
properly? j 



YES 



NO 



NA 



eOlttfENTS 
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TRAINING OBSERVATION WORKSHEET 
Name or Type of Training 



Unit- 



OBSERVATIONS 



YES 



NO 



NA 



COMMENTS 



13. 



14. 



Were training aids or materials used 
during this training? List those 
used. 



Were job aids handed, out during this 
training? Which ones? 



15. 
16. 
17. 
18. 

19. 

20. 
21. 

22. 
23. 
24. 



Did the training include use of the 
job aids? 



Did the training progress from expla- 
nation to demonstration to practice? 



j. 



Did the trainer answer the soldiers 1 
questions lO their satisfaction? 

Was the training site adequate from 
the standpoint of range facilities, 
space, support equipment and noise? 

it as the training site arranged so 
that the trainer could be seen and 
heard by all including yourself? 

Was the weather so bad as to 
distract from training? - 

For scheduled training, was the 
training time somewhat shorter or 
longer than the time allotted? 

Were there stated or written training 
objectives for this training? 

Was there a written or stated 
standard of performance? 

Did the training follow a trailing 
plan? Which one? 



/ 



25. Was a test heid? 



Training Elided (Time)_ 
Observer 



TRAINING OBSERVATION WORKSHEET 

Name or Type of Training 

Trainer (s) 



Date 



Unit- 



Training Site_ 



Training Objectives 

..." _ / 

1. Was the lesson based 
on stated or written 
obj ectives? 

2. Did they include ... 

- tasks? 

- : conditions? 

- standards? 

Lesson Presentation 

3. Did the trainer ... 

- review training 
objectives? 

■ ■— define new terms? 

- identify new 
equipment? 

- answer questions 
to students' 
satisfaction? 

- demonstrate tasks? 

Demons tra t ion 
U 9 eouid the soldiers .. 

- see the demo? 

- hear the demo? 

- follow the demo? 



Trainee (s) 



Training Started (Tiine)_ 



vrc 




XT A 


COMMENTS 


4 

Y 


N 


NA 








Y 


N 


NA 








v 


XT 


XI li. 

IMA 








Y 


N 


NA 








Y 


N 


NA 




♦ 


• 


Y 


N 


NA 








Y 


N 


NA 








Y 


n 










Y 


N 


NA 








Y 


N 


NA_ 








Y 


N 


NA 








Y 


N 


NA 
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TRAINING OBSERVATION WORKSHEET 


Date 


Unit 


Name or type of -^Training _ 












YES 


NO 


NA f 


COMMENTS 


Practice 










5. Did all soldiers ... 










separp-t; lyf 


Y 


3N 


NA 




- combine sufrtasks into 
whoie-t?:sk: practice? 


Y 


N 


NA 


j 


— nerf Drni ta^ks to 
standard unassisted? 


y 


N 


NA 


I 


6* Did an instructor ... 










- directly supervise 
each soldier's 










practice? 


Y 


N 


NA 




- teil soldiers what 
they were doing 
iif^Mt. ana wrong . 


Y 


N 


NA 




7* Did certain steps give 
soldiers more trouble? 




- N- 


NA 


: - ... 


rVllJ^L-IJL— O LC^ ».. »- - 


Y = 




f raiT^*! no T?nti ■{ *%m^n 1" 

and Materials 








i 


8. List any of the follow- 
xng useo uiung era in"* 
ing . • • 










— training equipment? 


Y 


N 


NA 




- training aids? 


Y 


N 


NA 




- job aids? 


Y 


N 


NA 




9. Did the training equip- 
ment work properly? 


Y 


N 


NA 





§0 
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TRAINING OBSERVATION WORKSHEET Date Unit^- 



Naine or Type of Training- 





YES 


NO 


NA 


eO^ENTS 


Did the training . . . 










— focus on the job 
aids? 


Y 


N 


NA 




- proceed from explan- 
ation to demonstra- 
tion to practice? 


Y 


N 


NA 




- follow a training 
plan? Which one? 


Y 


N 


NA 


/ ■ . . 


Training Environment 










Was the training site " 
adequate with regard 

tO r • • 








/ - 

7 


- range facilities? 


Y 


N 


NA 


1 
1 


- support equipment? 


Y 


N 


NA 






Y 


N 


NA 




* noise distractions? 


Y 


% 


NA 




Did discomfort due to 
poor weather distract 
from training? 


«! 


N 


NA 





Training Ended (Time)_ 
Observer 



si 



/ 
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ORGANIZATIONAL' MAINTENANCE OBSERVATION &*°RKSHEET 

Maintenance Sit e A, 

Time of Visit 



Date_ 
Unit 



Visit Announced 



Visit Unannounced 



OBSERVATIONS 



YES 



NO 



NA 



COMMENTS 



4. 



5. 



6. 



7. 



9. 
10. 

ii. 

12. 
13. 



Were any mechanics present in the 
motor pool? How many? 

Were any mechanics /Engaged in 
maintenance activities'? How many? 

Did you observe Mechanics engaged 
in any other ac/ivitie^? What 
activities? 



Was any maintenance being per-, 
formed on tjie Ml tanks in the \ 
motor pool? What was happening? 

Was there mar \ed damage to the 
external bodies of the Ml tanks? > 

Did you see copies of the -20 on 
the premises? 

Dijd the -20 f s show signs of use 
(elg., greasy fingerprints)? 

Were the tools in the tool bdxks 
clean? 

Were L tools being borrowed or 
shared among maintenance stations? 

Was the STE/M1 in use? 

Was there a separate tool room?' 

Was there a cleaning area for 
cleaning tools and parts with 
cleaning rags, solvents, etc;? 

Was maintenance supervision avail- 
able at the maintenance site? 



it 
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\ y\lil ?t\ SIGNAL MAINTENANCE OBSERVATION WORKSHEET . Date- 

.•a^itunance Site — ._ ' Unit 



OBSERVATIONS 


YES 


NO 


NA 


COMMENTS 














Were supervisors actively involved 
in the maintenance activities? 














15. 


•Was any training being conducted 
daring you :sit? 














16, 


Were work flow charts posted? 














17. 


Were there recent entries on the 
work flow charts? 














18. 


Were any crew members observed 
ordering parts from a 20 P manual? 


) 






4. 







Observat ion Ended (Time) — 



Observer 



4 



TESTING OBSERVATION WORKSHEET 
Name or Type oi Examination 
Examiner (s) — . 




Uriit- 



Examiriee(s) 



Testing Site- 



Test- 



Testing Started (Time)_ 



Retest- 



1 

NO NA 



OBSERVATIONS 

Did : examiner read the test instruc- 
tion to the soldiers? 

Were the test instructions stated 
clearly enough for the soldiers to know 
what they were expe: to do? 

Were pass/fail standards clearly ex- 
plained such that soldiers would know 
when they performed correctly? \ 

Were all tasks that were trained also 
tested? 

Were soldiers tested on any tasks that 
were not taught? 

Were the testing conditions the same 
as the training conditions? 

Were the standards used during 
training also used to score test 
performance? 

Did the examiner help or prompt the 
soldiers during the test? 



YES 



COMMENTS 



Testing Ended (Tirne)- 
Observer 
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